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Executive Summary 


This report presents results from the second phase of a descriptive study of the Researcher- 
Practitioner Partnerships in Education Research program. This two-year grant program, funded 
by the Institute of Education Sciences (IES) at the U.S. Department of Education, supports 
exploratory research within a partnership context. In each funded partnership, researchers 
collaborate with practitioners from state or local education agencies on a research project that 
investigates a problem of practice and identifies strategies to address the key issues. The National 
Center for Research in Policy and Practice (NCRPP), which is funded by IES, conducted the 
study. As a descriptive study, no inferences about the partnerships’ success or the program’s 
overall impact can be made. 


Research Design and Methods 

We studied the first three cohorts of researcher—practitioner partnerships (RPPs), funded in 
2013-2015, using a mixed-methods, cross-case design. A summary of Phase I findings can be 
found in NCRPP Technical Report No. 2, A Descriptive Study of the IES Researcher-Practitioner 
Partnerships Program. For Phase II, we developed two survey instruments, one for researchers 
and one for practitioners. The surveys included five previously-tested scales of items from 
NCRPP’s national survey of educational leaders’ research use as well as new items related to 
partnership goals, prior relationships, and future work together. New items were tested and 
revised through a cognitive piloting process. We also developed, pilot-tested, and implemented 
an interview protocol for each group, and we conducted a systematic document review of grant 
applications. 


A total of 114 participants completed the Phase II survey (response rate = 78%), including 62 
researchers (25 of whom were principal investigators) and 52 practitioners (28 of whom were 
co-principal investigators). Ninety-five of these individuals also participated in an interview 
(response rate = 65%) including 53 researchers (21 of whom were principal investigators) and 42 
practitioners (24 of whom were co-principal investigators). 


Context 

Each RPP in this study focused on a central educational issue, most often related to improving 
K-12 teaching and learning (12 of 27 RPPs). Three RPPs addressed issues of K-12 teacher 
quality or evaluation, and two centered on K-12 school improvement. Four RPPs identified early 
childhood education as their main issue, five pursued postsecondary access and success, and one 
RPP focused on improving coordination across state service providers and education agencies. 


The majority of RPPs focused their work on research questions that were descriptive or 
exploratory in nature, in accordance with the aims of the broader program. These projects sought 
to understand a particular education problem or issue, such as why a particular group of students 
was underperforming, and to identify possible intervention strategies. A few RPPs focused on 
understanding causal relationships or validating measures or constructs. Most partnerships used 
mixed-methods approaches that drew on both new and existing data sources. 
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The partnerships were given only two years of funding, and resources were not provided to set up 
or test interventions at scale. Rather, the purpose was to lay a foundation for future intervention 
research grounded in a more thorough understanding of the problem. 


Progress on Goals of RPPs 

The RPPs pursued a range of goals in their work together, including and extending beyond those 
emphasized in the program's request for applications (RFA). Partnerships reported that they 

were closest to accomplishing goals related to building a foundation of work together, followed 
by developing a deep understanding of the focal problem, researchers’ capacity to work in 
partnership, and a deep understanding of how researchers and practitioners can work together. In 
terms of growth over time, participants reported making progress on almost all goals, including 
those related to developing findings that apply to other organizations and improving students’ 
socio-emotional/non-cognitive outcomes. 


Perceived Benefits of Participating in a Partnership 

Researchers and practitioners alike highly valued their participation in partnership work, with 
almost all of those surveyed either agreeing or strongly agreeing that they would participate in 
another RPP in the future. 


RPP members reported that partnerships provided local policymakers with new ideas or 
frameworks or supported the design of professional development, programs, or practices. 
Participants from about one-third of partnerships reported that their work had contributed to a 
new or revised policy within the educational organization. 


Both researchers and practitioners contributed to the research effort and to dissemination. More 
precisely, both were involved in collecting, organizing, and analyzing data as well as presenting 
at conferences, including both researcher- and practitioner-oriented events. About half of the 
partnerships had members who had written for traditional research outlets (ie., articles, book 
chapters, or books) or who had contributed to new media platforms. 


Shifts in Researchers’ and Practitioners’ Engagement with Research and Practice 

On surveys, the majority of practitioners reported becoming better at using research in their work 
and were more likely to do so because of their participation in the partnership. Almost all of the 
researchers agreed that they had become better at conducting research that meets the needs of 
practitioners. Both researchers and practitioners agreed they would feel confident leading a future 
partnership. 


In interviews, participants further described practitioners’ increased appreciation for the value 
of research, their openness to participating in and using research, and their expanded skills 
related to developing, conducting, and disseminating findings from a research study. Likewise, 
researchers reported having developed expanded understandings about practitioners’ contexts, 
the value of their input in the research process, and the skills needed to adapt research methods 
and timely reports of findings to practitioners’ needs. Both groups noted that they had improved 
their skills in communicating with stakeholders. 
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Practitioners’ Use of Research 

The ways that RPP district leaders reported using research are similar to those reported by district 
and school leaders in a nationally representative sample. In both samples, educational leaders 
reported using research in multiple ways including to make decisions (instrumental use), to 
inform how they thought about issues (conceptual use), to persuade others of a particular point 
of view (symbolic use), or to integrate research processes into their own work (process use). 
Compared to the national sample, RPP district leaders reported less frequent symbolic use of 
research and more frequent process use. Within the RPP sample, practitioners in research roles 
were significantly more likely to report higher levels of process use of research than their peers in 
non-research roles. 


Among the activities research evidence might inform (i.e., instrumental uses of research), RPP 
practitioners were most likely to be involved in directing resources to a program, scaling up 

a program, or designing professional development. Although practitioners least frequently 
reported participating in purchasing an intervention or targeted program, they reported the 
highest frequency of research use for this activity. On average, RPP practitioners in non-research 
roles reported being more involved in activities related to purchasing an intervention or targeted 
program, redesigning a program, and designing professional development than did their 
colleagues in research roles. When these activities occurred, RPP practitioners reported that, in 
the past year, about half of their RPP research partners were involved in designing professional 
development or directing resources to a program. 


Useful Pieces of Research 

We asked RPP practitioners to name a piece of research that was useful to them, and we 
compared their reports to those of district leaders from a national survey with the same question. 
RPP practitioners most often named journal articles, whereas national survey respondents 
most often named books. Research named by RPP survey respondents focused on particular 
student subgroups more frequently than did research named by national survey respondents. 
RPP practitioners most frequently named pieces of research that focused on student learning 
and school organization but that did not have a disciplinary content focus. Finally, RPP study 
respondents noted that the piece of research they had named was useful because it helped 
with the design of programs, policies, and initiatives; national survey respondents, by contrast, 
frequently named reasons related to supporting leaders’ professional learning. 


Nature of Relationships Prior to the IES Grant 

The partnerships that received IES funding between 2013 and 2015 were not, for the most part, 
new collaborations. The majority had participants who had worked together before receiving IES 
RPP funding, and most participants knew at least one person in or had worked on a project with 
the partner organization before the grant started. Beyond these relationships, many partnerships 
had some infrastructure in place already. For example, 19 partnerships had established formal 
data-sharing agreements, 18 had established broader research agendas beyond the focus of the 
IES grant, 16 had established memoranda of understanding (MOUs), and 11 had established 
decision-making boards prior to receiving the IES grant. 
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Conditions for Starting and Maintaining a Partnership 

The top two conditions for launching an RPP were mutual organizational interest and trust 
among RPP members. Other conditions that supported starting an RPP included a data-sharing 
agreement or MOU, individual expertise of RPP members, and organizational leadership. 
Holding regular meetings, mutual organizational interest, and trust among RPP members were 
top conditions for maintaining a partnership. 


Promising Strategies for Overcoming Challenges 

RPP practitioners reported three main challenges in their partnerships: (1) turnover of 
positions for those involved in the partnership as well as leadership turnover within educational 
organizations more generally; (2) differences in researchers’ and practitioners’ typical timelines 
or pace of work; and (3) having the “right people at the table” in terms of active members in the 
partnership with decision-making authority to act on the partnerships’ findings. In interviews, 
RPP participants shared strategies they felt were useful in navigating these issues. The strategies 
included building strong, trusting relationships, communicating regularly, and being flexible 
enough to adjust course based on changing circumstances. 


Organizational Conditions in the Practice Organization 

Culture of research use. In terms of organizational culture for research use, the majority of RPP 
practitioners agreed that research was seen as a useful source of information in their organization, 
but fewer indicated that they were expected to back up claims with research in a meeting. Overall, 
RPP district leaders reported their organizational culture was less research-oriented than did the 
national sample subset of district leaders. 


Conditions that support learning in a partnership. Prior research suggests that some practice 
organizations may be better positioned than others to engage productively with their external 
partners. Specifically, internal communication may be a challenge in some practice organizations. 
Only half of RPP practitioners reported having enough time and space to make sense of new 
information from their partners or that new knowledge was regularly communicated across 
departments. A majority of practitioners reported that it was easy to see the connections between 
their organizations’ initiatives and work with external partners. However, two-thirds of RPP 
practitioners reported that organizational leadership did not coordinate work effectively enough 
to limit conflicts or reduce overlap between their organizations’ initiatives and partnership work. 
Finally, having the organizational resources (e.g., time, staff) to support partnering seemed to 
vary in practice organizations as well. 


Plans for Ongoing Work Together 

The majority of partnerships had continued working together past the end of the IES RPP grant 
or planned to continue to do so. Six of the 27 partnerships had successfully applied for and 
received additional funding; another five applied for additional funding but did not receive it. Ten 
ongoing partnerships had plans to apply for additional funding, while the remaining six did not 
have plans to apply for additional funding at the time of the Phase II survey. 
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Partnerships pursued future funding from the following organizations: IES, the Spencer Foun- 
dation, the National Science Foundation, the National Institutes of Health, local and national 
foundations (e.g., Annie E. Casey Foundation; James $. McDonnell Foundation; William T. Grant 
Foundation), and state agencies. 


Recommendations 

We offer some specific recommendations to the IES RPP program regarding the structure of 
the grant program, the support offered by IES during the application process, and the RFA and 
proposal guidelines. 


The program could consider a differentiated approach to RPP funding so that there are different 
goals, timelines, and funding amounts for newer partnerships compared to well-established ones. 
Further, IES may want to consider offering workshops for prospective teams to help them develop 
key skills related to RPPs. This may broaden the base of applicants and result in more successful 
new partnerships. 


In terms of the RFA and proposal guidelines, the IES RPP program may want to consider naming 
the range of short- and long-term goals that the partnerships have specified in the past, without 
limiting the possible goals that might be pursued. Further, many RPPs do integrate design into 
their plan of activities, but this could be encouraged explicitly in the RFA in order to reach impact 
on students more quickly. The grant application could ask for additional information related to 
the conditions that tend to support a partnership's launch or ongoing work, and applicants could 
be asked to offer initial ideas of how they might navigate common challenges, should they come 
up in the course of their work together. Finally, proposers could be asked to consider whether 
their list of participants includes those in the educational agency who have decision-making 
authority or involvement in implementation related to the problem of practice. 
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Introduction 


This report presents the results of a descriptive study of the Researcher-Practitioner Partnerships 
in Education Research Program, a two-year grant program funded by the Institute of Education 
Sciences (IES) at the U.S. Department of Education that supports exploratory research within 

a partnership context. In each of these partnerships, researchers collaborate with practitioners 
from at least one state or local education agency on a research project to investigate a problem of 
practice and to identify strategies to address the key issues. 


Since 2013, there have been five cohorts of partnerships funded through the grant program, 

and interest in research—practice partnerships (RPPs) in education has continued to grow more 
broadly.' New funding streams have emerged that support researchers and practitioners working 
in partnership with each other to address complex problems of practice (e.g., the William TT. 
Grant Foundation’s Institutional Challenge Grant;* the National Science Foundation’s CS4All 
initiative’; the Gates Foundation’s support for improvement networks*). Advocacy groups like 
Results for America and the Data Quality Campaign have pointed to RPPs as an important lever 
for supporting research use at district and state levels.° 


Resources for those who want to participate in partnerships have grown as well. There are 

several repositories with tools for starting or continuing partnerships online, including the 
Research+Practice Collaboratory® or the research-practice partnership microsite hosted by the 
William T. Grant Foundation.’ New books came out in 2018 that provide guidance for those 

who want to engage in equity-oriented partnerships.* The National Network of Education 
Research-Practice Partnerships (NNERPP)”’, an organization for RPPs, now has over 25 member 
partnerships. In addition to their annual forum, NNERPP curates a weekly blog on Education 
Week's website where RPP leaders share research findings and experiences from their partnership 
work.!° 


Yet we still do not have a good understanding of the range of ways researchers and practitioners 
can work together around a problem of practice. We need a better sense of the different ways 

to structure partnership work, the challenges that emerge, and the organizational and partner 
contexts that support the hard work of partnering. We also need to know more about the different 
goals that partnerships identify to orient their work together and the perceived benefits of their 
partnerships. From here, we can begin to understand when, and under what conditions, RPPs are 
productive strategies for instructional improvement efforts. 


This report presents results of the efforts of the National Center for Research in Policy and 
Practice (NCRPP) to address this need through a two-year descriptive study of researchers and 
practitioners engaged in RPPs funded by the Institute of Education Sciences. The NCRPP is an 
IES-funded center focused on the study of research use among education leaders in the United 
States. This is the second of two technical reports from the project." 


To develop an understanding of these RPPs, we surveyed and interviewed researchers and 
practitioners who were actively involved in the partnerships and analyzed their original grant 
applications. 
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In this report, we present the results of our study with regard to the following topics and 
questions: 


About perceived benefits of partnerships: 


e What progress did RPPs make on their identified goals over time? 

¢ What were the perceived benefits of the partnerships? 

¢ How did participation in an RPP contribute to shifts in how practitioners engaged with 
research? 

¢ How did participation in an RPP contribute to shifts in how researchers did their 
research? 

¢ How did RPP practitioners use research, and what types of research did RPP practitioners 
find useful? 


About partnership contexts: 


e What was the nature of RPP members’ relationships prior to their IES grant? 

e What were the primary conditions that supported starting and maintaining a partnership? 

¢ What promising strategies for overcoming key challenges were identified by RPPs? 

¢ What organizational conditions that support research use and partnership work were 
present in the practice organizations? 

e What were RPP members’ plans for future work together after their IES grant concluded, 
including their pursuit of funding opportunities? 


In this report, we describe our instrument development process, sampling strategy, and key 
constructs. We then offer analysis for the questions above. It is our hope that findings of this 
study can not only inform the IES RPP program but also contribute to knowledge on the 
processes, successes, and challenges of RPPs in education. This work provides information about 
the reported value of these collaborative efforts for researchers and practitioners interested in 
developing partnerships. 
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Conceptual Framework 


A set of key interlocking ideas guided the design of this study. We drew on Weiss and Buculavas’s 
typology of research use” to address the question of how well grantees are meeting a core goal 
of the IES RPP program: building educators’ capacity for using research. To inform the design 

of survey and interview questions related to goals, we drew on a recently developed framework 
developed by Henrick and colleagues" for characterizing the outcomes of research-practice 
partnerships. Finally, to provide insight into organizational dimensions related to research use, 
we drew on ideas of culture for research use and organizational conditions that may support an 
organization's ability to productively engage with a partner. 


Multifaceted Nature of Research Use 

There are different ways that research can be involved in decision making. When policymakers 
and others encourage school and district leaders to use research, they often imply that leaders 
should use research directly and centrally to make decisions related to policy or practice. 
However, research can also influence decision making by focusing attention on issues that were 
previously unknown to decision makers," identifying opportunities for improving current 
programs and policies,'* or providing information about the plausibility of policy theories of 
action.'® 


In designing queries about leaders’ purposes for research use, we followed the categories 

first identified by Weiss and Buculavas and applied more recently in studies of educational 
leaders’ research use.'” This typology suggests that leaders’ use of research is multifaceted and 
characterized by at least four main roles for research:'® 


¢ Instrumental use: Research is applied to guide or inform a specific decision. 

¢ Conceptual use: Research induces changes in the way a person views either a problem 
or the possible solution space for a problem. 

¢ Symbolic/political use: Research is used to validate a decision or legitimate a decision 
already made. 

e Process use: Leaders incorporate the processes of research into their own work, for 
instance, launching an evaluation study, participating in a grant proposal that includes 
an evaluation component, or collaborating with others to analyze data. 


Research-practice partnerships are hypothesized to promote use of all types because they provide 
opportunities for sustained interactions between researchers and practitioners around evidence." 
Sustained interaction is important to fostering research use because it involves the interactive 
processes of deliberation, persuasion, negotiation, and sensemaking.”” RPPs frequently involve 
structured activities to develop research and evaluation questions together and to make sense 

of results of studies of policies and programs.”! Such processes may not only help practitioners 

to make sense of evidence, they may also be occasions for researchers to “give sense”” to the 
meaning of the evidence in light of how particular findings fit in with other research studies. 
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Characterizing the Goals of IES Researcher-Practitioner Partnerships 

Promoting research use is not the only goal of RPPs, nor is it the only goal of the IES researcher- 
practitioner partnerships grant program. For example, many RPPs work together to design 

and test interventions that can improve learning outcomes and be implemented at scale.”* To 
characterize the goals of RPPs in this study, we draw on a recent effort by Henrick and colleagues 
(funded by the William T. Grant Foundation) to characterize the different goals pursued by 
RPPs.” This effort, which used an iterative, participatory process of soliciting input and feedback 
from multiple RPPs across the country, yielded a typology of five categories of outcomes that the 
RPPs in the sample agreed were important to them to varying degrees. Though this framework 
encompasses some aims that might go beyond those explicitly promoted in the IES RPP program, 
we employed it here in order to capture the various purposes of the participating partnerships. 
We describe below the five categories of outcomes from the framework. 


Support improvements to teaching and learning. RPPs support educational partner organizations 
to achieve their local improvement goals. That is, the research they do is in service of larger aims 
for improving teaching and learning outcomes rather than just to develop an understanding of 
problems. Sometimes, partnerships work together to identify and test strategies for addressing a 
focal problem. Partnerships may also engage in continuous improvement research in which they 
develop, test, and refine particular strategies and use research evidence to refine or adjust those 
strategies.” Researchers in partnerships may develop indicators or indicator systems to help 
partners track progress toward their own program goals.”° 


Conduct and use rigorous research. Just as researchers who engage in other forms of research and 
development do, researchers in RPPs aim to conduct research that meets the highest standards of 
quality. When evaluating programs, for example, they seek to use the most appropriate designs for 
estimating the causal impact of programs available, including (but not limited to) experimental 
design.”” What distinguishes RPPs from other forms of research is that they are focused sharply 
on their local partners’ identified problems of practice. To that end, partnerships often produce 
descriptive studies that explore the relationships among malleable factors in educational 
environments,” as is one aim of the IES RPP program. Partnerships are also consumers of 
research. Sometimes, education leaders take up findings and use them to adjust policies and 
programs.” On other occasions, design teams composed of both researchers and practitioners 
make use of research analyses to refine their solutions to educational problems.* As part of this 
goal, practitioners’ organizational capacity to conduct or use research may be further developed. 


Inform the work of others. Although RPPs attend to local problems of practice, most also seek to 
inform the work of others outside the partnership. Researchers can contribute to new knowledge 
and theory that furthers our understanding of what it takes to support educational improvement 
across different educational settings.*! When given the opportunity, educators readily share 
knowledge, tools, and practices they have developed with other educators outside their own 
organization.» There is evidence, too, that such tools and practices can be taken up by educators 
in other partnerships.*? 
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Cultivate partnership relationships. RPPs aim to cultivate partnership relationships and build trust 
among researchers and practitioners. Productive working relationships are both the foundation 
for joint work and a consequence of working successfully together. Similarly, trust is a necessary 
ingredient in all partnerships,™ and it develops as people make commitments to one another and 
follow through on those commitments. 


Increase capacity of researchers and practitioners to conduct partnership work. Another goal that 
RPPs pursue in order to sustain future work is to develop members’ skills and dispositions to 
conduct work in partnerships. For researchers, this includes identifying problems of practice 

to study that could also address gaps in foundational knowledge about learning or education. 
Likewise, it includes cultivating dispositions to listen to practitioners and seek out their expertise 
in diagnosing problems and designing solutions to them. For educators, it means cultivating an 
appetite for research evidence and developing skills necessary to participate in different aspects 
of the research process, from defining questions to providing feedback on instruments to making 
judgments about how best to apply findings in particular decision-making contexts. 


Organizational Conditions that Support Research Use and Partnership Work 
Research suggests that certain organizational conditions may be related to patterns of research 
use.** Here we describe these key conditions. 


Culture of research use. One condition that may matter is the culture of research use within 

the educational organization. A culture of research use is one in which organization members 
value research as a resource for decision making,** remain open to change in light of evidence,” 
and enact multiple social supports and norms promoting evidence use.** NCRPP’s nationally 
representative survey of district and school leaders found that higher reported levels of 
organizational culture for research use were positively associated with greater reports of all types 
of research use.” Further, structured opportunities to engage with others around research within 
a partnership can be associated with greater research use within decision making.” 


Organizational conditions to engage productively with an external partner. Not all educational 
organizations are equally positioned to act on the knowledge they gain by working with research 
partners. Past research suggests that an organization's capacity to do so is related to its “absorptive 
capacity”—that is, its ability to recognize the value of new information, assimilate it, and apply 

it in novel ways as part of organizational routines, policies, and practices.’ There are three 
conditions that can support an organization's absorptive capacity: communication within and 
between departments; resources to support partnering work; and leadership activities to connect 
partnership work to organizational goals or to manage the activities of multiple partners.*” When 
present, these conditions may make it more likely that the organization is able to apply new 
knowledge and ideas gained from working with an external partner in productive ways. 
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Study Design 


This report summarizes a two-year, two-phase study. This descriptive study focused on the 

IES Researcher-Practitioner Partnerships in Education Research Program grantees’ prior 
relationships and future work, reported goals and progress, challenges and strategies, conditions 
that supported and fostered partnership work, attitudes toward working in partnerships, and 
RPP practitioners’ reports of research use and organizational conditions in their agencies. As 

a descriptive study, no inferences about the program's impact can be made, nor was that the 
intent. Below, we describe our study population and sample, study methods, instruments, and 
procedures for data collection and analysis. 


Study Population and Sample 

We studied the first three cohorts of the IES RPP program, with two-year funding beginning 

in 2013, 2014, or 2015. (For more information about the RPPs in these cohorts, see the next 
section, “Descriptions of the IES Researcher-Practitioner Partnerships.”) The population included 
a total of 28 RPPs and the most active researchers and practitioners in those partnerships. By 
“most active” members, we mean those with some responsibility for carrying out the work of the 
partnership, as identified by each RPP’s principal investigator (PI). 


“RPP researchers” or “researchers” refers to the PIs and other RPP members from universities 
and research organizations; this group does not include practitioners who were in research 
positions located in practice organizations (e.g., a director of research in a school district). “RPP 
practitioners” or “practitioners” refers to co-principal investigators (co-PIs) and other RPP 
members from educational organizations such as school districts, departments of education, and 
social services agencies (including individuals in research-related roles in these organizations). 
Each partnership had one researcher PI and either one or two practitioner co-PIs. 


This project unfolded over two years: Phase I (summer 2016) and Phase II (summer 2017). Below 
we describe the rostering and sampling for both phases.* 


Phase | 
Original roster. In order to identify the population of participants for the study, we began with the 
publicly available abstracts of the 28 RPPs. We aimed to include three to four active participants 


per RPP, with roughly equal researcher and practitioner representation, including the PI anda 
co-PI. 


In May 2016, we emailed the Pls of all 28 RPPs to invite participation, to confirm the names and 
email addresses of the active members we had identified through publicly available abstracts, to 
request the names and email addresses of active members we had not identified, and to request a 
copy of their grant application. Of the 28 PIs, 27 responded to our requests. 


The PIs confirmed or named 82 active researchers (including the 27 PIs) and 78 active 
practitioners (including 25 lead co-PIs), for an overall total of 160 individuals rostered for Phase 
I. In the case of four RPPs where the PI named more than one co-PI, we asked them to identify a 
lead practitioner co-PI. 
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For each of the responding 27 RPPs, our roster included the PI, the lead co-PI, an average of two 
other researchers, and an average of two other practitioners. 


Participation and response rates. In Phase I, rostered individuals received up to six email 
invitations and reminders to participate, as well as a telephone call as a second-to-last reminder. 
Individuals were asked to first complete a 15- to 20-minute survey, after which they were 
automatically prompted to schedule a 45- to 60-minute interview with a member of the research 
team. 


As shown in Table 1, a total of 104 participants representing 27 RPPs completed the survey, the 
interview, or both, for an overall response rate of 66% and an average of four participants per 
RPP. Participation for 25 of the 27 RPPs included at least one researcher and one practitioner. We 
received survey responses from 104 participants (response rate = 65%) and conducted interviews 
with 98 participants (response rate = 61%). Eight participants completed the survey but not the 
interview. Two participants completed the interview but not the survey. (These two individuals 
started but did not fully complete the survey items, yet still received an automatic invitation to 
schedule an interview.) 


Response rates in Phase I were higher for researchers than for practitioners. Of the 82 rostered 
researchers, 61 (74%) completed the survey, the interview, or both, while 45 (58%) of the 78 
rostered practitioners did so. Among researchers and practitioners, participation was high for 
PIs (26; 93%) and for co-PIs (21; 84%), and lower for other researchers (35; 65%) and other 
practitioners (24; 45%). 


Table 1. Phase | Roster and Response Rates, by Role in Partnership 


# of participants 
(response rate) 


Survey, interview, or 


Rostered Survey Interview both 
104 98 106 
All 160 (65%) (61%) (66%) 
56 61 
Researchers 82 ee! (68%) (74%) 
26 24 26 
Pls Ze (93%) (86%) (93%) 
Others 34 hee nee ee 
Practitioners 78 nie oo so 
Co-PI Practitioners 25 a oy oe 
Others 53 ay cae ren 
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Phase II 

Original roster. To begin Phase II of the study, in February of 2017 we confirmed with IES the 
closing dates of all 28 RPP grants. For the nine grants that had closed in 2016, the Phase II roster 
included the same RPP members as on the Phase I roster. For the other 19 grants that had not yet 
closed, we emailed each PI to confirm that Phase I rostered members were still active in the RPP, 
to ask if there were any changes to active membership, and to request the contact information for 
any newly active RPP members. 


Of the 82 researchers rostered in Phase I, 73 were still active members in Phase II (or were part 
of grants that had closed) and remained on our Phase II roster. Likewise, of the 78 practitioners 
rostered in Phase I, 68 remained on our Phase II roster. A total of 141 individuals therefore were 
included on both Phase I and Phase II rosters. 


In addition, two newly active researchers and three newly active practitioners were added to the 
Phase II roster. This resulted in a roster of 75 researchers (including the same 27 PIs as in Phase 

I) and 71 practitioners (including 31 co-PIs), for a total of 146 individuals on the Phase II roster. 
In one of the RPPs, the PI had been replaced by a researcher who was not active in Phase I. In one 
other RPP, the PI role shifted from the original PI to another researcher on the project, but both 
were active in both phases of the study. 


In an attempt to improve practitioner response rates, in Phase II we mailed to rostered 
practitioners a $10 gift card with the initial invitation to participate. Otherwise, we proceeded 
with a similar process of up to six email and telephone invitations and reminders to complete the 
15-minute survey, after which participants received an invitation to sign up for a 45- to 60-minute 
interview. 


Participation and response rates. In Phase II, we secured participation from the same 27 of the 
28 funded RPPs. As shown in Table 2, of the 141 individuals who were rostered in Phase I and 
remained on the Phase II roster, 86 participated in Phase II, including 49 researchers and 37 
practitioners. 


Table 2 details the full Phase II roster, including newly active RPP members who were added and 
overall participation in Phase IL. A total of 114 participants completed the survey, including 62 
researchers (25 of whom were PIs) and 52 practitioners (28 of whom were co-PIs). Ninety-five of 
these same participants also participated in an interview, including 53 researchers (21 of whom 
were PIs) and 42 practitioners (24 of whom were co-PIs). 


Response rates in Phase II were notably higher than in Phase I, with 83% of researchers and 73% 
of practitioners participating, for an overall response rate of 78% among the 27 rostered RPPs. As 
in Phase I, there was an average of four participants per RPP in Phase II. At least one researcher 
and one practitioner participated in the Phase II survey in all 27 RPPs, and at least one from each 
of the two groups participated in the Phase II interview in 24 of the 27 RPPs. 
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Table 2. Response Rates for Pls, Other Researchers, Co-Pls, and Other Practitioners 


Both Phase I| and Phase II Phase II 


# of participants 
(response rate) 


Rostered Participated Total Rostered Survey Interview 
114 95 
All 141 86 146 
(78%) (65%) 
62 53 
Research 78 49 i 
esearchers (85%) (71%) 
25 21 
PI 26 24 sd 
s (93%) (78%) 
37 32 
Oth 47 25 8 
ers (77%) (67%) 
52 42 
Practiti 68 37 va 
ractitioners (73%) (59%) 
28 24 
Co-PI Practiti 29 28 31 
O ractitioners (90%) (77%) 
24 i 
Oth 39 9 a 
ers (60%) (48%) 


Participant characteristics. Table 3 below details characteristics and demographic information 
for the 114 participants in Phase II. About two-thirds of researchers worked at universities, with 
the remaining at other research organizations; most were either professors or researchers in 
these settings. Almost two-thirds of practitioners worked in school districts, with the remaining 
in departments of education, higher education organizations, or other educational or social 
services agencies. The largest number (22 of 52; 42%) were in research-related roles across P-20 
educational organizations, with others in a variety of roles within early childhood, K-12, and 
postsecondary organizations. 


Researchers and practitioners were experienced in their positions and typically held advanced 
degrees. On average, PIs had served 14 years in their positions, compared to about nine years 
served by other researchers and co-PI practitioners. Other practitioners averaged five years in 
their positions. Almost all PIs (96%) and other researchers (81%) held doctoral degrees. Notably, 
over half of co-PIs (57%) and 42% of other practitioners also held doctoral degrees. Across all 
groups, participants largely identified as female (61%) and White or European American (76%). 
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Table 3. Characteristics and Demographic Information of Survey Respondents in Phase II, by Role in 


Partnership 
Researchers Practitioners 
All Pls Others Co-Pls Others 
n 114 25 37 28 24 
Type of Research Organization 
University 40 16 24 - - 
Research Organization 22 9 13 - - 
Type of Practice Organization 
School District 32 - : 19 13 
Department of Education 9 - - 6 3 
Higher Education Organization 6 - - 2 4 
Other Educational or Social Services 
Agency : " : 1 ‘ 
Researcher Role in Organization 
Professor 32 17 15 - 7 
Researcher 25 7 18 : = 
Doctoral Student 3 0 3 = = 
Center Director 2 1 1 - : 
Practitioner Role in Organization 
fen y Assessment, and 22 - - 14 8 
Early Childhood Policy and Practice 8 - - 3 5 
Postsecondary Policy and Practice 6 - - 3 3 
K-12 Federal Programs 5 - - 2 3 
K-12 Deputy Superintendents and 3 - - D) 1 
Chief Officers 
K-12 Special Education 3 - - 1 2 
K-12 Educator Evaluation 2 : = 2 ) 
K-12 Curriculum and Instruction 2 - - 1 1 
Average years in current role 99 14.4 8.6 9.5 5.0 
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Table 3. Characteristics and Demographic Information of Survey Respondents in Phase II, by 
Role in Partnership (continued) 


Researchers Practitioners 
All Pls Others Co-Pls Others 
n 114 25 37 28 24 
Highest degree held 
Doctorate 72% 96% 89% 50% 39% 
Master’s 25% 4% 9% 45% 57% 
Bachelor’s 3% 0% 3% 5% 4% 
Gender identity? 
Female 66% 58% 66% 55% 87% 
Male 34% 42% 34% 45% 13% 
Racial or ethnic identity 
White or European American 83% 81% 89% 85% 74% 
Latino(a)/Hispanic 6% 4% 0% 15% 9% 
Sos ee ee 
Asian American or Pacific Islander 4% 4% 9% 0% 0% 
Multiracial 2% 4% 3% 0% 13% 
Arab American or Indian American 1% 4% 0% 0% 0% 
American Indian or Alaskan Native 0% 0% 0% 0% 0% 
Multiracial 0% 0% 0% 0% 0% 
Other 1% 4% 0% 0% 0% 


aGender identity was asked as an open-ended question. 


Sources of Data 
Our analysis in Phase II drew on survey data and interviews. We describe these data sources 


below. 


Survey. We developed two survey instruments, one for researchers and one for practitioners. 
Surveys included five previously tested scales of items from NCRPP’s national survey of 
educational leaders’ research use and attitudes, including instrumental, conceptual, and symbolic 
uses of research, as well as new items specific to this study. New items were tested and revised 
through a cognitive piloting process with two practitioners and two researchers. We also solicited 
expert feedback from advisors. We then made revisions to items on the basis of team discussions 
that took into account the pilot results and expert feedback. The constructs below are those 
included in this report, listed in the order of the subsequent findings sections. 
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Our surveys provided respondents with a definition of research as “an activity in which people 
employ systematic, empirical methods to answer a specific question.” The survey text also 
provided the following elaboration: 


Research bases its conclusions in investigations involving statistical data, interviews, 
observations, and case studies, or a combination of these. Research can appear in 
books, academic journal articles, practitioner-oriented journals, and analyses of 
program implementation developed by researchers external to the district. It can also 
appear in policy and evaluation reports or presentations developed by researchers 
within a district. 


For this study, we differentiate between research, which involves systematic inquiry 

to answer a specific question, and the practice of looking at data from the district, 
school, or classroom, which is more open-ended and seldom addresses specific 

research questions. For instance, looking at state standardized test results to identify 
students who need extra support in the classroom would not be research. However, 
asking the question, “What is the relationship between fourth-grade state standardized 
test results and high school graduation?” would be research. 


Goals. To elicit respondents’ reports on partnership goals, we first provided a list of 14 potential 
goals, based on the RPP goals framework developed by Henrick and colleagues. Goals included 
plans to identify a specific strategy for improvement, improve students’ academic outcomes, or 
build a foundation or infrastructure for future work together. In the Phase I survey, we asked 
whether this was a current goal of the partnership, was a goal at the start of the partnership, or 
was not a goal. If a respondent indicated this was or had been a goal, we subsequently asked what 
progress the partnership had made towards that goal. Then, in the Phase II survey, we asked 
participants to report on their progress on the goals they had identified in the first survey. Item 
response choices were: 1 = no progress; 2 = a little progress; 3 = some progress; 4 = accomplished; 
or 5 = exceeded. This set of items was included on both researcher and practitioner surveys. 


Perceived benefits of partnership: Attitudes. We included items focused on attitudes towards 
partnerships. These items included statements like, “I would participate in a researcher- 
practitioner partnership in the future,’ or negatively phrased statements such as, “I would not 
recommend to a colleague that they join or form a researcher-practitioner partnership.” We asked 
respondents to indicate the extent to which they agreed or disagreed with each statement, using 
these item responses: 1 = strongly disagree; 2 = disagree; 3 = agree; 4 = strongly agree. This set of 
items appeared on both researcher and practitioner surveys. 


Perceived benefits of partnership: Involvement in research dissemination and local practices. On 

the Phase II survey, researchers and practitioners were asked to indicate their involvement in 

a variety of activities related to conducting research and disseminating research findings. One 
bank of items focused on their involvement in research activities, with statements such as, “I have 
participated in collecting, organizing, or analyzing data as part of this partnership” 
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The other bank of items focused on the contributions of the partnership’s work, with statements 
like, “Our IES RPP has contributed or helped lead to the design of professional development (PD) 
related to the focal problem(s) of the partnership.” On each of these items, respondents marked 
“yes” if the statement was true for them and/or their partnership. 


Research use: Instrumental. Instrumental use occurs when research is applied to guide or inform 
a specific decision. To elicit practitioners instrumental use of research, we provided a list of six 
decisions common to educational organizations, including curriculum adoption, scaling up 

a pilot program, designing PD, and other activities. This scale was adopted from the NCRPP 
research use survey where it demonstrated excellent reliability (a = .93). Because this study 
surveyed practitioners in a range of educational organizations (not only school districts), we 
omitted one item that asked about involvement in adopting curriculum materials. We also 
condensed two items on designing PD for administrators and teachers to one item on designing 
PD. We first asked participants if they had been involved in each type of decision. Those who 
indicated they were involved in an activity were then asked how often they had used research 

as part of that activity. We also asked practitioners whether the partnership was consulted or 
involved in the activity. Item response choices were: 1= never; 2 = sometimes; 3 = frequently; 4 = 
all of the time. This item bank only appeared on the practitioner version of the survey. 


Research use: Conceptual. Conceptual use refers to research changing the way a person views 

a problem or space of possible solutions. In our survey, we sought to elicit the extent to which 
research informs practitioners’ ways of looking at problems or their approaches to solving district 
problems. ‘The six-item scale included questions such as, “How often have you encountered 
research that changed the way you look at problems facing your school/district?” and “How often 
have you encountered research that suggested alternative solutions to a district problem?” This 
scale was adopted from the NCRPP research use survey, where it demonstrated good reliability (a 
= .88). As with the instrumental use scale, item response options were: 1= never; 2 = sometimes; 
3 = frequently; 4 = all of the time. This item bank only appeared on the practitioner version of the 
survey. 


Research use: Symbolic. Symbolic use, sometimes referred to as political use of research, occurs 
when research is leveraged to influence a decision or to legitimate a decision that has already 
been made. The four-item scale asked respondents to report their engagement in activities such 
as using research to mobilize support for important issues or selectively using research to support 
decisions. This scale was adopted from the NCRPP research use survey, where it demonstrated 
good reliability (a = .81). Item response choices were: 1 = never; 2 = sometimes; 3 = frequently; 4 
= all of the time. This item bank only appeared on the practitioner version of the survey. 


Research use: Process. Process use occurs when leaders incorporate the processes of research 

into their own work. The four-item scale asked respondents to report their engagement in 
activities such as launching an evaluation study, participating in a grant proposal that includes an 
evaluation component, or collaborating with others to analyze data. Item response choices were: 
1 = never; 2 = sometimes; 3 = frequently; 4 = all of the time. This item bank only appeared on the 
practitioner version of the survey. 
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Prior relationships. Participants were asked to describe the nature of their prior relationships with 
RPP partners based on a number of indicators. First, they were asked whether they or anyone 

in their organization had worked with members of their partner organization(s) in any capacity 
before the IES RPP grant started, either formally or informally. Those who indicated that a prior 
relationship existed between partner organizations were asked: (1) to indicate whether they 
personally had any relationships with anyone from the partner organization(s); (2) how many 
projects the partner organization(s) had worked on together prior to the start of the IES RPP 
grant; and (3) how many years the partner organization(s) had worked together prior to the start 
of the IES RPP grant. Respondents who had established a relationship were also asked to indicate 
whether any of the following structures were in place prior to the start of the IES grant: (1) a 
memorandum of understanding (MOU); (2) a data-sharing agreement; (3) a research agenda that 
was broader than their IES RPP grant’s focus; or (4) a formal body that makes decisions (e.g., an 
advisory board). These items appeared on both researcher and practitioner surveys. 


Conditions for launching or supporting a partnership. During Phase I of our study, interview 
participants talked about conditions that supported the launch and maintenance of their RPP. 
Based on the main patterns in interviews, we generated a list of nine supportive conditions 
(plus an “other/please specify” option) that we included on our Phase II survey. These 
conditions included holding regular meetings, mutual organizational interest, and support from 
organizational leadership, among others. We asked all survey respondents to rank order the top 
three conditions that supported the launch or maintenance of their partnership. 


Organizational culture of research use. During both phases of survey administration, practitioners 
were asked to respond to a set of four items related to the culture of research use within 

their educational organization. This scale was adopted from the NCRPP research use survey, 
where it demonstrated good reliability (a = .87). These items included statements like, “In my 
organization/district, research is seen as a useful source of information” and “In my organization/ 
district, it is expected that if you make a claim at a meeting, you will be able to cite research 
evidence to back it up.” We asked respondents to indicate the extent to which they agreed or 
disagreed with each statement, using the following item responses: 1 = never; 2 = occasionally; 3 
= often; 4 = all of the time. These items were only asked of practitioners. 


Absorptive capacity: Communication, strategic knowledge leadership, and resources. During both 
phases of survey administration, practitioners were asked to respond to three sets of items related 
to their organizational capacity to learn from their work with partners. One of these item banks 
included statements related to intra-organizational communication. These statements included, 
“In my organization/district, ideas, information, and new knowledge are regularly communicated 
across departments,” as well as negatively phrased statements such as, “It’s difficult for me to find 
out what's going on in other departments in my organization/district” Practitioners were asked 
to respond to a set of items related to resources needed for partnership activities, with statements 
like, “My organization/district has the discretionary funding needed to devote to the planning 
and implementation of partnership activities.” Finally, practitioners responded to a set of items 
related to the leadership of partnership work, with statements like, “My organization/district 

has the capacity to coordinate efforts with all of our external partners.” We asked respondents to 
indicate the extent to which they agreed or disagreed with each statement, using the following 
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item responses: 1 = strongly disagree; 2 = disagree; 3 = agree; 4 = strongly agree. These items were 
only asked of practitioners. 


Future plans. In order to determine the future plans of each participating partnership, we first 
asked survey participants to indicate whether or not their IES RPP grant was still in progress. 

If their grant had already ended, respondents were asked whether or not the partnership had 
continued working together, and if they had applied for or received additional funding in order 
to continue this work. If the grant was still in progress, respondents were asked whether or not 
the partners had made plans to continue the work once the grant ended, and whether they had 
applied for or planned to apply for additional funding. In both cases (i.e., grant completed or 
still in progress), if respondents indicated that additional funding had been or would be sought, 
we asked them to identify the names of the grant-funding institutions. If survey respondents 
indicated that their partnership’s work did not or would not continue beyond the life of the 
grant, we asked them to choose all the applicable reasons why from a list of options that included 
differences in priorities among RPP organizations, changes to the agendas of RPP organizations, 
and insufficient broader institutional support for the work, among others. Items related to 
partnerships’ future plans appeared on both researcher and practitioner surveys. 


Demographic information. Finally, we asked participants to report their own demographic 
characteristics related to years of experience, educational degrees earned, gender, and race or 
ethnicity. 


Interview protocols for researchers and practitioners. We developed two semi-structured interview 
protocols, one for researchers and one for practitioners, both of which we piloted before 
beginning the interviews. Each protocol included questions related to the partnership's goals 

and progress made toward those goals, how the partnership compared to previous experiences 
of interacting with researchers or practitioners, challenges to sustaining the partnership and 
strategies, and perceived benefits of the partnerships. 


Data Collection Procedures 

We administered the survey via Qualtrics, an online survey administration platform, with an 
eight-week completion window for each respondent. We sent an email to each contact on the 
Phase II roster with an invitation to participate in the study and a link to the appropriate survey 
on Qualtrics. We sent three reminder emails over three weeks and followed up with one to two 
phone calls to survey non-respondents. In some cases, we delayed the reminder if we learned that 
the respondent was unavailable for periods of time over the summer. 


Surveys took 15 to 20 minutes to complete. If a participant started the survey, they automatically 
received an email with a link to schedule a phone interview. Interview scheduling involved the 
same pattern of three emails at weekly intervals and one to two phone call reminders. Therefore, 
unless a person directly declined to participate, he or she could receive up to eight emails and 
four phone calls over the 10-week period to invite participation in the survey and interview. After 
that point, we determined that no response was a signal for non-participation. Ninety-five of the 
114 survey respondents scheduled an interview with one of the members of the research team. 
Interviews lasted 45 minutes, on average. 
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We aimed to have both researcher and practitioner perspectives represented in as many RPPs as 
possible. For 24 of the 27 participating RPPs, we secured a minimum desired response of at least 
one researcher and one practitioner for each RPP for both the survey and interviews. In the three 
remaining RPPs, we secured the minimum desired response for survey participation but not for 
interviews. In these cases, only practitioners from the RPPs participated in an interview. 


Issues of Confidentiality and Anonymity 

Our reporting here focuses on broad themes and patterns that emerged across partnerships, not 
on individual partnerships. As such, in this report we do not identify any individual partnerships. 
We have likewise avoided using any identifiable language related to the individuals or the 
partnerships. 


Approach to Analysis 

To answer our research questions, we first looked at each source of data separately, producing 
descriptive statistics from our analysis of surveys and coding summaries of interview data. We 
considered carefully which data source was best suited to help us describe different aspects of the 
partnerships. Below, we describe the analyses of proposals, surveys, and interview transcripts. 


Closed-ended survey analysis. After survey data cleaning of closed-ended items, one research team 
member calculated descriptive statistics for reported survey constructs. In order to investigate 
agreement between groups within partnerships, we compared responses among PIs, co-Pls, other 
researchers, other practitioners, all researchers, and all practitioners. To investigate similarities 
and differences across RPPs, we determined whether at least 50% of each RPP’s participants gave 
consistent responses for items or banks of items. There were some instances where we reported 
other groups (e.g., PI/co-PI; practitioners only) when we felt that a particular group was ina 
better position to report on a topic. We note this throughout the report when appropriate. 


Open-ended survey analysis. In the survey we asked practitioners to “Think about a time when a 
piece of research you encountered changed your thinking or opinions about possible solutions to 
your organization’s/district’s problems. What was that piece of research?” We asked respondents 
to provide as much detail as possible, including title, author, year published, publisher (if known), 
topic, and how it had been useful to their work. The following definition of research appeared 
above the item: 


For the purposes of this study, “research” is an activity in which people employ systematic 
methods to answer a specific question. Research, as we define it, is different than the 
practice of looking at data from the district, school, or classroom, which is more open- 
ended and seldom addresses specific research questions. 


Two members of our team located the research named by respondents and coded each piece for 
APA citation, author, summary/abstract, format, type of review, topic (up to two), subtopic (up to 
two), content area focus (up to two), student subgroup focus (up to three), and how it was useful. 
The research team then summarized counts for each category for all respondents. We considered 
patterns for practitioners who worked in school district roles in order to compare them to the 
results of a national survey of school and district leaders. 
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We also analyzed patterns for those who worked in research roles, since a substantial proportion 
of IES RPP practitioners worked in research departments or had research roles in the practice 
organization. 


Interview analysis. Interviews were audio-recorded and transcribed, and research team members 
reviewed the transcripts for accuracy, replaced individuals’ names with pseudonyms, and 
uploaded cleaned transcripts to Dedoose, a qualitative research software program. Team members 
tagged each transcript with descriptors for the data collection time period (summer 2017); 
participant type (practitioner, researcher, PI, or co-PI); and RPP (by number) in order to track 
individuals and partnerships. 


We proceeded in an iterative fashion to analyze the interview data. First, we engaged in low- 
inference coding. The goal of this initial stage of coding was to index interviews by key constructs 
like “challenges.” Lead researchers created a coding guide with major constructs, definitions, 
relevant interview questions, examples, and non-examples for each code. This coding guide 

was revised as the team coded transcripts, first together and then independently once they had 
reached a shared understanding of codes. A subset of four team members engaged in this initial 
coding. Lead team members periodically checked coding to ensure integrity to the coding guide, 
and the team met regularly to discuss any issues that arose. 


Next, for each specific line of analysis, lead team members conducted a more refined analysis of 
coded data to identify themes. Starting with the relevant coded excerpts, team members began 
with an a priori, deductive coding list based on the current literature base summarized in our 
conceptual framework above. They added codes inductively based on themes that emerged from 
the data. Team members reconciled coding for the same interviews until they reached at least 
70% agreement, then proceeded to meet weekly to review and discuss questions about their 
individual coding. We analyzed patterns from interview coding by summarizing counts that 
showed predominant patterns and variations and used memos and matrices to consider patterns 
by individual, role group, and partnership. 


Study Limitations 

One limitation to both phases of this study is that we have relied on retrospective self-reports 
from surveys and interviews. Self-reports of socially desirable behaviors, like research use, 

are always subject to bias. Further, the fact that these self-reports are retrospective means that 
respondents may have glossed over challenges, dilemmas, and uncertainties in decision making. 
Research use and the role for partnerships are interactive phenomena that are best understood 
using multiple methods, including observations of partners’ work together and the role of 
research therein. 


Second, only a year elapsed between our two data collection points. It is possible that partnerships 
were able to accomplish a number of their short-term goals in this year, but it may have been 

too short of a time frame to see major changes. Further, a cohort of 11 partnerships had already 
completed their IES grants in the summer of 2016. In these cases, there was no new partnership 
activity sponsored by IES between 2016 and 2017, although many of these RPPs continued their 
work together. Their reports of progress towards goals, therefore, reflect progress not directly 
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supported by IES funding. Where it made sense, we compared patterns by cohort, and we report 
any differences in the sections that follow. 


Further, as a descriptive rather than evaluative study, we cannot make inferences about the im- 
pact of the IES RPP partnerships. The data we gathered provide us with self-reports of different 
constructs, but different study designs would be needed to track other questions (e.g., long-term 
impact on student outcomes). Finally, the original IES request for applications (RFA) emphasized 
the research equally to the development of the partnership. This report does not address the re- 
search itself or the quality of the research. 
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Description of IES Researcher-Practitioner Partnerships 


In Table 4 below, we provide an overview of the 28 partnerships that were funded through the 
IES RPP program and began in 2013-2015. All information presented here was gathered from 
publicly available sources. 


The 27 RPPs that participated in our study had been awarded two-year grants in the first three 
years of the IES RPP program, with six RPPs beginning in 2013, 13 in 2014, and eight in 2015. 
Due to no-cost extensions granted to some RPPs, 11 of the partnerships ended in 2016, and the 
remaining were scheduled to complete their grants in 2017 or 2018. 


Table 4. Descriptions of the First Three Cohorts of IES RPPs 


RPP Grant Title Partners Topic Level Description 


Boston Public American Insti- Extended learn- K-12 This partnership sought 

Schools Expanded tutes for Research ing time (ELT) to track and categorize 

Learning Time and Boston Public how much ELT was 

Research Schools added to BPS schools, 

Collaborative how that time was used, 
and the impact and cost 
analysis for 


implementation. 
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Table 4. Descriptions of the First Three Cohorts of IES RPPs (continued) 


RPP Grant Title Partners Topic Level Description 
What Works for Arizona State School K-12 The partnership proposed to 
Title | Schools: University andthe improvement identify measures in exist- 
Understanding the Arizona Depart- strategies ing data in order to assess 
Contributors and ment of Education the effectiveness of school 
Barriers to School improvement strategies 
Improvement and factors that predict 
student achievement within 
priority secondary Arizona 
schools. The results of the 
partnership were intended 
to inform future research 
proposals to properly assess 
the effectiveness of 
promising school 
improvement strategies. 
Applicants at the Center for Teacher hiring K-12 This researcher- practitioner 
Doorstep: Education Data partnership aimed to study 
Improving Hiring and Research the relationship between 
Practices through a_ at University of teacher performance and 
Better Washington and data collected during the 
Understanding of | Spokane Public teacher hiring process, 
the Link Between — Schools including observations by 
Applicant local hiring officials. The 
Information and partnership hoped to better 
Teacher Quality understand the relationship 
between applicant data 
and teacher performance in 
order to create a 
foundation from which to 
develop a future research 
proposal which would 
develop and analyze the 
effectiveness of new 
teacher applicant 
assessment tools. 
Career Pathways Chicago Citywide Adult education, Postsec- _ The goal of this 
Programming for Literacy Coalition, career pathways ondary partnership was to assess 


Lower-Skilled 
Adults and Immi- 
grants: A 
Comparative 
Analysis of Adult 
Education 
Providers in 
High-Need Cities 


Houston Center for 


Literacy, Institute 
for the Study of 
Adult Literacy at 
the Pennsylvania 
State University, 
and Miami-Dade 
County Public 
Schools 


how adult education 
providers in each city 
integrated career pathway 
components into their 
services. They sought to 
understand commonly used 
student outcome measures, 
the short- and long-term 
outcomes of adult education 
providers, and how provid- 
ers design and implement 
career pathways. 
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Table 4. Descriptions of the First Three Cohorts of IES RPPs (continued) 


RPP Grant Title Partners Topic Level Description 
Foundation for Connecticut State | Educational K-12 This partnership aimed to 
Alliance for Department of services for collect data on court-in- 
Education Education, the court-involved volved youth, develop a 


Providence Public 
Schools District 
and Education 
Development 
Center: Developing 
a Researcher- 
Practitioner 
Partnership to 
Improve 
Achievement 
among Minority 
Students 


Improving 
Paraprofessionals 
Instructional and 
Behavioral Support 
In Urban 
Elementary School 
Settings: A 
Research 
Practitioner 
Partnership 


Connecticut youth 
Judicial Branch 

Court Supported 

Youth Services 

Division, and Yale 
University’s Child 


Study Center 

Education English language K-12 
Development learners 

Center and 


Providence Public 
Schools District 


Juniper Gardens Paraprofession- K-12 


Children’s Project, _ al practices, 
University of training, and 
Kansas, Kansas evaluation 


City, and Kansas 
Public Schools 


blueprint for a cross- 
agency partnership, and 
use the research findings 
to inform policy and 
practice regarding the 
delivery of appropriate 
educational services for 
approximately 1,500 
juveniles in detention and 
10,000 other court-in- 
volved youth. 


This partnership aimed to 
foccus on collaborative 
research to inform policies 
and programs designed to 
improve English 

Language Learner learning. 
This included developing a 
database that would 
catalog available data 
elements related to ELL 
students including demo- 
graphic characteristics, 
English language aquisi- 
tion, achievement on state 
assessments, and ELL 
program enrollment. 


The goal of the 

partnership was to eval- 
uate the knowledge, 
responsibilities, 
implementation of 
effective instructional 
practices and better under- 
stand the training needs 

of para-professionals and 
the teachers who work 
with them. The partnership 
aimed to collect 
observational data for the 
purpose of creating, imple- 
menting, and 

evaluating a training 
system for para- 
professionals and their 
supervising teachers. 
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Table 4. Descriptions of the First Three Cohorts of IES RPPs (continued) 


RPP Grant Title Partners Topic Level Description 
New York City New York City College readiness K-12, This researcher- 
Partnership for Department of Postsecondary practitioner partnership 
College Readiness | Education, the City aimed to build capacity 
and Success University of NY, for improving 
and the Research postsecondary education 
Alliance for New opportunities for New 
York City Schools York City students by 
at New York shedding light on the 
University high school to college 


transition and creating 
conditions for developing 
new interventions that 
aim to improve student 
preparation. The 
partnership proposed 

to conduct exploratory 
studies to develop col- 
lege-readiness indicators 
that incorporate indi- 
vidual, community, and 
organizational factors 
that enhance or inhibit 
college preparation and 
then develop systematic 
interventions that 
enhance these factors. 


Exploring Early Oregon Social Early childhood Early The partnership aimed to 
Childhood Learning Center, education education help the district collect 
Education United Way of information about early 
Experiences and Lane County, and childhood education and 
School Adjustment South Lane School child care experiences of 
in Rural Elementary District incoming kindergarteners 
School Students in order to examine 


associations between 
these experiences and 
school readiness, and ac- 
ademic and social-emo- 
tional adjustment for the 
general population and 
high-risk sub- 
populations (ELLs, SES, 
rural). 
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Table 4. Descriptions of the First Three Cohorts of IES RPPs (continued) 


RPP Grant Title Partners Topic Level Description 
The Oregon Oregon State English language CUZ The purpose of the partner- 
English Learner University and learners ship was to determine the 
Alliance: A Oregon Depart- best practices to support 
Partnership to ment of Education academic achievement 
Explore Factors among current and former 
Associated with ELLs. The partnership aimed 
Variation in to enhance existing 
Outcomes for databases to include 
Current and information on ELL status, 
Former English build a model for 
Learners in Oregon characterizing 
instructional programs, and 
examine the relationship 
between instructional 
program and educational 
outcomes of both current 
and former ELLs. 
A Researcher- SRI International English language K-12 The partnership aimed to 
Practitioner and Clark County _ learners, science further develop and design a 
Partnership to School District education theory of action to promote 
Promote English the academic success of 
Language Learners’ ELLs in science by 
Science Learning organizing evidence of 
in the Elementary research-based strategies 
Grades and interventions. The 
partnership sought to study 
patterns of variations in 
ELLs' performance and 
explore the relationships 
between ELL's science 
scores and other factors. 
Raising GPA: Strategic Education Socio-emotional Kel The partnership aimed 


Partnering to 
Increase Grit, 
Perseverance, 
and Achievement 
in Baltimore City 
Middle Schools 


Research 
Partnership (SERP), 
Baltimore City 
Public School 
System (BCPSS) 


learning 


to measure persistence 

and self-efficacy in school 
climate assessment. Then, 
the partnership planned to 
investigate school-based 
processes that might 
contribute to social- 
emotional learning skills and 
develop effective 

strategies for improving 
student perserverence in 
sixth grade classrooms with- 
in the Baltimore City public 
schools. 
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Table 4. Descriptions of the First Three Cohorts of IES RPPs (continued) 


RPP Grant Title Partners Topic Level Description 
Creating a The Collaborative Socio-emotional K-12 The partnership aimed 
Monitoring System for Academic, learning to collect socio-emo- 
for School Districts Social, and tional learning data from 
to Promote Emotional Learning students and teachers 
Academic, Social, (CASEL) and to develop reliable and 
and Emotional Washoe County valid teacher and student 
Learning: A School District self-report measures of 
Researcher- SEL skills. The partnership 
Practitioner would study how different 
Partnership SEL dimensions were 

associated with academic 
outcomes and conduct 
analyses to determine if 
SEL skills served as pro- 
tective factors that could 
moderate student risk. 
Designing a RCT The RAND Community Postsecondary The partnerships sought 
Experiment to Test Corporation and college to conduct preliminary 
the Impact of In- the Texas Higher preparation research to determine the 
novative Interven- Education effectiveness of 15 
tions and Policies | Coordinating Board developmental education 
for Postsecondary programs instituted 
Developmental Ed- in eight higher public 
ucation: ARAND— education institutions 
TX Higher Educa- across Texas. They aimed 
tion Coordinating to investigate whether 
Board Research developmental education 
Partnership programs for first-time 
students enrolling in the 
state's public communi- 
ty college system were 
effective for students and 
cost-effective for the state. 
Academic The San Diego On-track indica- K-12 This RPP sought to 


Trajectories and 
Policies to Narrow 
Achievement Gaps 
in San Diego 


tors for student 
performance 


Education 
Research Alliance 
at the University 
of California San 
Diego and the 
San Diego Unified 
School District 


enhance to the 

district's access to higher 
quality student data there- 
by producing ideal student 
trajectories as well as 
accurate on-track 
indicators of individual stu- 
dents that can be shared 
with school staff, parents, 
and researchers. This work 
could then inform the 
district's plans for 
interventions for 

students most in need of 
assistance. 
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Table 4. Descriptions of the First Three Cohorts of IES RPPs (continued) 
RPP Grant Title 
The School District The University 


Description 
The goal of this 


Partners Topic Level 
School/district K-12 


of 
Philadelphia-Penn 


of Pennsylvania 


improvement 


Graduate School of strategies 


partnership was to 
build a structured, 


Graduate School of Education and the formalized relationship 
Education School District of between the partners 
Researcher- Philadelphia to build the capacity 
Practitioner of both organizations, 
Partnership in improve research-to- 
Education practice links, study the 
Research district's approaches 
to school reform, and 
establish foundations 
for future, larger-scale 
collaborative research. 
California University of Community college Postsecondary The goals of the 
Community California Davis and career partnership were to 


Colleges and Ca- 


and California 


technical education 


identify students that 


reer Technical Community enter community 
Education: A College colleges with clear 
Research- Chancellor's Office career technical 
Practitioner (CCCCO) education aims and 


Partnership 


Implementing 
Comprehensive, 
Integrated, Three- 
tiered Models to 
Meet Students’ 
Academic, 
Behavior, and 
Social Needs: A 
Researcher- 
Practitioner 
Partnership 


University of 
Kansas, Arizona 
State University, 
and Lawrence 
Public Schools 


Implementation of 
behavioral preven- 
tion 


K-12 


CTE programs that lead 
to financially 

rewarding 
employment, as well as 
policies and 

practices that promote 
student success in CTE 
programs. 


This RPP aimed to 
evaluate the newly 
installed district-and 
school-level 
implementation of 

the comprehensive, 
integrated, three-tiered 
(CI3T) model of 
behavioral prevention. 
The partnership sought 
to evaluate the CI3T 
model with four studies 
focusing on treatment 
integrity, social 

validity, screening, 
teacher efficacy, and 
future professional 
development needs. 
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Table 4. Descriptions of the First Three Cohorts of IES RPPs (continued) 


RPP Grant Title Partners Topic Level Description 
Blended Learning —_ University of Blended K-12 The goals of this 
at Scale- Maryland at learning partnership were to 
Implementation College Park and examine the relationship(s) 
and Analysis of District of between blended 
Student Columbia Public learning 
Achievement in Schools implementation and 
District of students' ELA achievement 
Columbia Public on the DC Comprehensive 
Schools Assessment System exam 
as well as identify key 
conditions and practices 
related to blended 
learning implementation 
that may be associated 
with improved student 
achievement. The 
partnership findings will 
directly inform the dis- 
trict's strategic decisions 
concerning blended 
learning. 
Miami-Dade University of Early childhood Early This partnership aimed 
County Partnership Miami, education education to examine the school 
for School Miami-Dade readiness skills and early 
Readiness and County Public school success of chil- 
Early School Suc- Schools, the dren who transition from 
cess Children's Trust, a range of early care and 
and the Early education settings into the 
Learning Coalition Miami-Dade County public 
of Miami-Dade/ schools. It would involved 
Monroe a needs assessment and 
descriptive analyses of 
school readiness data. 
Students in Foster University of Foster care K-12 The partnership aimed to 


Care: The 
Relationship 
between Mobility 
and Educational 
Outcomes 


Northern Colorado, 


Colorado 
Department of 
Education, and 
Colorado 
Department of 
Human Services 


placement and 
school out- 
comes 


track the 

residential and 
educational stability and 
academic outcomes of 
Colorado students in fos- 
ter care as well as identify 
how changes in foster care 
placement and school sta- 
bility were related to each 
other through analysis of a 
joint human-services/ 
education database. 
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Table 4. Descriptions of the First Three Cohorts of IES RPPs (continued) 
RPP Grant Title 
META Researchers 


Partners Topic Level Description 


University of Teachers' data use K-12 This partnership aimed to 


and Practitioners in 
Partnership (RPP) 
to Enhance Data 
Use Practice that 
Improves Student 
Learning 


Virginia and 
Chesterfield 


County, Hanover 


County, Henrico 
County, and 
Richmond City 
Public Schools 


collect data concerning 
K-12 teachers’ current 
data use to inform a pilot 
professional development 
model which will be 
examined in year 2 to 
assess its influence on 
teachers' data use and 
associations with 
improved student 


learning. 

Examining the University of Teacher K-12 The goal of this 

Effects of IMPACT Virginia and Dis- effectiveness partnership was to 

on Students trict of Columbia understand the effects 

Achievement: Public Schools of IMPACT, the DCPS 

DCPS-UVA Effectiveness 

Research Assessment System for 

Partnership School-based Personnel, 
on student achievement, 
and with this information, 
guide part of its ongoing 
design refinement. The 
partnership also sought to 
investigate the 
differences across time in 
teacher applicant 
characteristics, teacher 
mobility, and aspects of 
teaching that were most 
malleable in context of 
strong incentives. 

The Seattle University of School discipline K-12 The purpose of the the 

Minority Washington and partnership was to build 

Engagement and Seattle Public district capacity to assess 

Discipline Schools and address 

Reduction disproportionality in 

Research school discipline practices 


Collaborative 


among minority student 
bodies as well as negative 
student outcomes as a 
result of disciplinary 
actions. 
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Table 4. Descriptions of the First Three Cohorts of IES RPPs (continued) 


RPP Grant Title Partners Topic Level Description 
A Partnership to Washington State Literacy K-12 The purpose of this RPP 
Improve the Use of University and outcomes was to conduct preliminary 


a Developmental 
Assessment 
Framework in 
Kindergarten 


Exploring 
Longitudinal 
Outcomes and 
Trajectories for 
English Language 
Learners (ELOTE) 


Creating a 
Connecticut Early 
Childhood 
Education 
Research Alliance 


Washington State 
Education Services 
District 105 


WestEd and Fort 
Worth 
Independent 
School District 


Yale University 
School of 
Medicine, 
Education 
Development 
Center, and 
Cooperative 
Education Services 


education 


English language K-12 
learners 


Early childhood Early 


education 


research on the adequacy of 
TSGold for assessing literacy 
skills and the potential of 
WakKIDS in improving 
literacy outcomes. 


The purpose of this 
partnership was to study 
the academic trajectories 
and outcomes of secondary 
immigrant English Language 
Learners in order to inform 
district policies, improve 
instructional and 
extracurricular programs, 
and refine classroom 
practices to better meet the 
needs of ELLs. The 
partnership sought to 
understand the academic 
trajectories of ELLs based on 
several factors which include 
ELL status, English language 
proficiency, test scores, and 
attendance to inform the 
implementation of promising 
programs and practices for 
secondary immigrant ELLs. 


The goals of this 
partnership were to 
understand how children 
perform on state-developed 
preschool measures, how 
preschool child outcomes 
predict kindergarten out- 
comes, and what teacher 
and school-level factors 
contribute to preschool and 
kindergarten performance. 
These findings would then 
inform interventions and 
adjustments to school cli- 
mate and teaching 
practices that impact 
preschool performance. 
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Progress on Identified Goals 


In Phase I, we found that the RPPs were pursuing a range of goals in their work. These goals fit 
within the stated goals of the IES RPP program, as they focused on developing practitioners’ 
capabilities to conduct or use research and explore a problem through rigorous research activities. 
Other goals surfaced as well, including goals to cultivate partnership relationships, support 
improvements to teaching and learning, and inform the work of others. Here, we share the 
progress that the RPPs reported making on these goals in Phase II. 


Key Findings: 
e RPP participants reported making progress on almost all goals they had identified. 


e RPP participants were asked to report progress along five dimensions. They made the 
most overall progress on goals related to building a foundation of work together, followed 
by developing a deep understanding of their focal problem, improving researchers’ 
capacity to work in partnership, and developing an understanding of how researchers and 
practitioners can work together. 


e Participants also reported progress on goals related to supporting improvements in 
teaching and learning and developing findings that apply to other organizations. The 
progress on these goals is noteworthy, given that these are likely lagging indicators and are 
thus expected to take more time to accomplish. 


In Phase I of data collection, we drew on survey and interview responses to identify the 
partnerships’ main goals. We found evidence that the IES RPPs pursued multiple goals that 
reflected Henrick and colleagues’ RPP outcomes framework described in an earlier section. In 
Phase II, we gathered information on the RPPs’ self-reported progress toward meeting these same 
goals. Specifically, we asked each participant what progress the partnership had made toward each 
type of goal they had indicated on the Phase I survey that they were pursuing. Similarly, we asked 
each participant in the Phase II interviews to tell us about the partnership's progress on the goals 
they had named in Phase I. 


Here, we summarize participants’ progress on their goals—as they reported on surveys—in line 
with the five dimensions of Henrick et al’s framework: (1) supporting improvements to teaching 
and learning; (2) conducting and using rigorous research; (3) informing the work of others; 

(4) cultivating partnership relationships; and (5) increasing capacity to conduct partnership 
work. For each dimension, we draw on interview data to offer more detail on the progress of 
partnerships toward their goals. 


As in other sections in this report, these are self-reports. As such, these findings should not be 
interpreted as indicators of success for a particular partnership or the RPP program as a whole. 


Progress on Goals between Phase | and Phase II 
Figure 1 shows progress reported by participants in both phases of the survey on the five 
dimensions named above. Overall, there were small positive increases for almost all goals. 
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There was relatively more progress made for goals related to supporting improvements to 
teaching and learning and informing the work of others. However, these goals lagged behind 
other foundational goals that showed more overall progress.* 


Develop deep understanding of focal problem 


Identify a specific strategy for improvement 


Develop approaches to evaluate 
an improvement strategy 


Improve organizational processes 
to impact students' outcomes 


Improve students’ academic outcomes 


Improve students’ social-emotional/ 
non-cognitive outcomes 


Make progress toward addressing 
focal problem through research activities 


Develop evidence for publication 
in a peer-reviewed journal 
Develop organization’s capacity 
to use research 


Develop findings that apply to 
other organizations 


Develop understandings of how researchers 
and practitioners learn from each other 
Develop researchers’ capacity 
to work in partnerships 


Build foundation for future work together | 3.9 | oa 
1 2 3 4 5 
m Support Improvements to Teaching and Learning » Conduct and Use Rigorous Research 
» Inform the Work of Others m Cultivate Partnership Relationships 


™ Increase Capacity to Conduct Partnership Work 


Figure 1. Self-reported Phase | to Phase II progress on specific goals, clustered by dimension. Across both 
surveys, n = 87 (50 RPP researchers, 37 RPP practitioners). 1 = no progress; 2 = a little progress; 3 = some 
progress; 4 = accomplished; 5 = exceeded. Lighter shading in each bar represents reported progress in 
Phase |; darker shading at end of bar represents the shift in Phase Il. For instance, on “Improve students’ 
academic outcomes,” Phase | average was 2.0 and Phase II showed a 0.3 increase, for an overall average of 
2.3 across the two phases. 


Technical Report No. 3 36 
National Center for Research in Policy and Practice 


Supporting Improvements to Teaching and Learning 

All 27 RPPs had a member who, in Phase I, identified supporting improvements to teaching 

and learning as an important goal of the partnership. Again, there were high levels of agreement 
within partnerships, as 21 partnerships had at least one researcher and practitioner who discussed 
their improvement efforts in Phase I. 


Relative to the other dimensions, less overall progress was reported for goals related to improving 
teaching and learning outcomes, such as improving PD or improving student outcomes. This is 
not surprising, as these important outcomes likely follow from the foundational development 

of relationships and trust necessary for joint work. Even so, there was reported growth in 
progress toward meeting this goal despite the relatively short time period of the IES RPP grant. 
Participants reported the greatest shift for the goal of improving students’ academic outcomes, 
which shifted from 1.9 at Phase I to 2.3 at Phase II (2 = a little progress, 3 = some progress). 


We see the most reported progress for developing a deep understanding of the focal problem (an 
average of 3.6, where 3 = some progress, 4 = accomplished). Initial progress here makes sense for 
two reasons. First, this goal was highlighted in the original IES RFA, and it was present in all of 
the original RPP applications. Second, any additional work together to design or implement new 
tools or practices would likely be predicated upon a deeper understanding of the problem. 


Next, partnerships might iteratively identify strategies for addressing the problem and then 
implement and study them. It is at this point that we might begin to expect to see improvements 
in teaching and learning. Some RPPs reported progress in these areas, and they attributed this 
progress to their co-design of successful tools or supports for teachers to change their practice. 


One partnership exemplified this underlying development progression towards local 
improvement goals: 


Year 1 was learning what we didn't know, getting familiar with the context, and 
working with the district partners to understand what problems we were actually 
going to try and solve. We took a year doing it. We did a bunch of different 

data collection techniques and approaches, drawing on improvement science to 
guide some of that work. 


Then, Year 2 was our first effort to take what we learned in Year 1, and have teachers 
take that up and do some improvement work across five different elementary schools. 


This year, in our no-cost extension year, we have sat back and figured out what 
worked and what didn't on two levels. One, did we accomplish our process goals 
in terms of getting teachers to try out and iteratively refine new strategies 

over time? Second, what about the focal practice, the content of what teachers 
did with students—what was useful and not from Year 2? What can we do going 
into Year 3 that we can learn from and apply? 
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From there, we learned, it’s positive evidence. Is it conclusive in any way? It’s not 
enough to write a paper about, but it’s enough to try something out for another 
school in our local context. We're not trying to say that these strategies 

are good for all kids in all time zones yet. 


Over their three-year grant period, this RPP was able to make progress on short-term and 
long-term goals in interconnected ways. It started with a deeper understanding of the problem 
informed by ideas of improvement science but soon transitioned into supporting changes in 
teachers’ practice through PD, coaching, and tool development. The partnership was careful not 
to jump ahead to causal claims about impacting student outcomes, focusing instead on replicating 
their model within the local context. However, both practitioners and researchers were hopeful 
about the positive shifts they were seeing in student dialogue, the focus area of the project. 


In another case, the partnership developed a longitudinal dataset of indicators relevant to 
kindergarten readiness, connecting early childhood and K-12 data systems for the first time in 
their local context. This enabled them to identify specific areas for improvement and plan for 
professional learning to those ends. A practitioner in the RPP explained: 


We have a good three-year dataset of knowing what preschool kids know 
and what they don't know coming into kindergarten. We have a good sense 
of where we want to go in terms of professional development, essentially 
outlining what are the expectations of a kindergartener, so preschools 
know that. Then the third big goal that we've been working on is providing 
professional development to preschool providers around early literacy and 
early math skills to get kids ready more quickly. 


This partnership first developed the foundation needed to gather information about and 
understand the focal problem and then used these data to plan for implementation to address 
identified needs. In this way, progress on supporting improvements to teaching and learning may 
have involved more foundational dimensions of other partnership outcomes described below. 


Conducting and Using Research 

Participants from all 27 RPPs described goals related to conducting and using research to address 
their focal problems and reported progress towards these goals between Phase I and II. On the 
Phase II survey, respondents reported, on average, a response of 3.5 (where 3= some progress and 
4= accomplished) towards addressing their focal problem through research activities. 


In terms of conducting research, partnerships made progress on their research goals by creating 
multi-sector datasets, conducting secondary analyses of student-, teacher-, and school-level data, 
creating case study reports, conducting interviews and focus groups, and more. One practitioner 
described how qualitative data collection in the first year then shaped the quantitative approach 
in the second year: 


In the first year of our partnership, it was qualitative. We were doing a lot of 
interviews, and ... we asked a question: What does successful extended learning 
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time look like in your school? From that question, that’s what we used to create our 
quantitative portion. ... This year, we’ve been looking at what data the district 
has that could fit those categories [that emerged from the interview responses]. 


We also saw evidence of RPPs’ progress towards building practitioners’ capacity to engage with 
research in ongoing ways. For instance, one practitioner reported on how the district’s culture 
had shifted in terms of school leaders and board members asking more often for data or research 
when making decisions. The practitioner attributed this shift to the influence of the RPP. He 
explained: 


It's more common for the school board to ask, “Well, what does the data say?” 
[and] for the schools to ask, “What’s the data say?” That kind of mentality 
around data has changed in our district and in our community. 


In another case, a researcher told us that he thought experiences with the partnership would help 
his practice partners ask researchable questions in the future. He offered: 


There was capacity building through all the conversations we had about why 
we use particular methods. [The practice partners] had not been able to doa 
matching study before. We really walked them through what that entailed and 
what that meant. ... That way of thinking about it was a useful exercise for them, 
and that will probably lead them to ask some good questions in the future when 
they're setting up studies. 


Based on comments like these, it appears that practitioners’ participation in research activities in 
turn increased their own capacity to conduct and use research in the future. 


Informing the Work of Others 

The RPPs were not just concerned with solving local problems or in sharing research findings 
with their local partners. In 25 RPPs, at least half of the participants agreed that developing 
findings that applied to other organizations was a goal of their partnership in Phase I. On average, 
the RPPs that had this goal reported making progress here as well (an average of 3.5, where 3 = 
some progress, 4 = accomplished). 


Many of the participants reported sharing findings in a variety of formats. These included 
traditional outlets for sharing findings, like peer-reviewed journal articles, research briefs, book 
chapters, or books. Partnerships also engaged in non-traditional dissemination strategies, like 
launching a publicly available website, facilitating webinars, and writing blogs, as well as attending 
and presenting at both practitioner-oriented and researcher-oriented conferences. (For more, see 
the next section, “Perceived Benefits of Participating in a Partnership.”) 


For example, one practitioner described developing a video based on the partnership's 
experiences: 
We tried to be deliberate about using our funds in a way that would allow us 
to share what was learned because we were only working with a few schools at 
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that time, and it wasn’t clear how much we might want to scale the learning 

or the implementation. We took the opportunity to create some videos to 
document the resources that were developed, how teachers from our district 
were involved, the identification of the intervention, the implementation, and 
some of the adjustments that were made. The video also highlighted our students’ 
perspectives on the intervention and its effectiveness. 


Finally, many partnership participants noted that they would continue to work with their data 
even after the grant was over, as it provided a rich data source. 


Cultivating Partnership Relationships 

All 27 partnerships named cultivating partnership relationships as a goal for their work together 
over the life of the grant, and the RPPs seemed to make the most progress in this area. Survey 
respondents reported an average of 3.8 (where 4 = goal was accomplished) for building a 
foundation for future work together. 


Initially, partnerships tended to foundational needs such as identifying individuals to play key 
roles in the partnerships and dedicating staff to RPP efforts. However, establishing deep, trusting 
relationships was not something that could be achieved at a single point in time, early on, nor 
could it be taken for granted in all future work together. Instead, relationship building required 
ongoing, continuous attention. 


Often, relationships needed to be cultivated when a partnership brought new stakeholder groups 
into the fold of the work. For example, one researcher described how their partnership brought 
new participants on board over the two-year period of the grant: 


The first year of the project, we worked on establishing relationships with 
teachers and the administrators at the school, getting into classrooms, and 
making people comfortable with the notion that a coach would be in and 

out of classrooms. This year, we've gone a bit deeper. This year, [the coach] 
was working at all of the schools and attended the learning community 
meetings, helped with assessments, did co-teaching and team teaching. ... 

So, one of the big changes was the deepening relationship between [the coach] 
and the teachers in the schools and their increased ability to use data to 
determine student needs and then work with her to craft lessons to meet 
those needs. 


Relationship building was not only a goal in and of itself. Often, cultivating and managing 
trusting partnerships was necessary to advance other goals of the partnership work. In the case 
above, developing these deep connections at the school site enabled the partnership to support 
improvements to teaching and learning at the school sites. Likewise, a foundation of trust helped 
partners to navigate issues they faced as they pursued their goals together. 


Technical Report No. 3 40 
National Center for Research in Policy and Practice 


Increasing Researchers’ and Practitioners’ Capacity to Conduct Partnership Work 
Across the two years of our study, participants reported high levels of overall progress in 
developing researchers’ and practitioners’ capacity to work in partnership as well as in developing 
a deep understanding of how researchers and practitioners can work together. The average rating 
was 3.6 in Phase 2 (where 3 = some progress on goal, 4 = accomplished goal). 


In many partnerships, participants discussed how their work with their RPP partners was more 
collaborative than it was in other research engagements. For instance, one researcher described 
how the researchers and practice partners worked together through all phases of their project: 


We've finalized our data collection, we've analyzed data, and we have put together 
presentations—all of those pieces we did in collaboration with our partners who 

are on the ground. For example, when we're talking about data collection, they 
would help us organize. They really have been key in helping us establish ourselves, 
our credibility, why people would want to work with us. ... In the data analysis, we 
will put together our findings and they would review them and give them feedback 
or ask questions, to make us think a little more deeply about what we've been doing 
—so, from a different perspective. ... In presenting, we would put a report together 
and then send that out and get feedback, clarification, interpretation of our findings, 
as well as insights or telling us, “You know, this is not gonna fly when we present it to 
the field. You might want to clarify this.” 


This example demonstrates how the partnership engaged in data collection, analysis, synthesis, 
and dissemination in a collaborative and iterative way. The insights from all parties were viewed 
as central and important to the research process and enhanced the quality and utility of the 
products that emerged as a result of this work. In addition to stating how much they valued these 
collaborative efforts for this project, participants expressed the hope that they would be able to 
work in these new ways in the future. 


Implications 

Consistent with the purpose of the IES RPP program, nearly all partnerships hoped to develop 
practitioners’ capabilities to conduct or use research or to explore problems through rigorous 
research activities. These were not the only goals that partnerships pursued, however. The RPPs 
also had explicit goals to cultivate partnership relationships, to increase capacity to conduct 
partnership work, to support teaching and learning, and to inform the work of others. This 
pattern of results indicates that the IES RPP program supports its stated goals of building 
partnerships and developing deeper understandings of pressing problems in the practice in 
education. 


In line with Henrick et al’s framework, the reported progress presented here provides evidence 
that there are likely “leading” and “lagging” indicators for partnerships’ progress on their goals. 
Initially, partnerships need to focus on building relationships of trust and develop an understand- 
ing of what researchers and practitioners bring. Not surprisingly, partnerships reported the most 
progress in the two areas of building a foundation for future work and developing the capacity for 
joint work. Building a foundation for work together may be a leading indicator that we might 
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expect to be accomplished sooner than other goals. Lagging indicators—e.g., shifts in student 
outcomes—may be expected to take more time to accomplish. 


Further, in our analysis of Phase I and II data, we saw increases in the reports of progress for al- 
most all goals identified, despite the relatively short one-year time frame between survey adminis- 
trations. The greatest shifts between Phase I and II occurred for the “lagging” goals—for example, 
improving students’ socio-emotional outcomes. So, even though these lagging goals were not fully 
accomplished by the end of this study, it is noteworthy that participants reported incremental 
progress here. We suspect there may be a general developmental trajectory for partnerships’ prog- 
ress on their goals, where developing a strong foundation lays the groundwork for subsequent 
progress on new policies or practices that contribute to improvements in student outcomes. 


It may also be the case that these goals are interdependent in partnership work. ‘That is, while 
building a foundation for work together may be a goal that demonstrates progress earlier in the 
partnership, it may be pursued jointly with and through other goals related to increasing capaci- 
ty, conducting and using research, supporting teaching and learning, and informing the work of 
others. Those looking to track the progress of RPPs may want to consider how and when these 
goals are independent and when they may be related. For example, one research participant sum- 
marized the three equally-weighted goals of their partnership this way: “The first basket would 
be that the research is completed, the second would be the nurturing of the relationship through 
the collaboration structures, the third is the capacity building for our partner through our inter- 
actions.’ A future study could try to understand how these three disparate but clearly connected 
goals advance the overall progress of the partnership. 


It is important when considering the progress of these partnerships to recall that their grants 
provided only two years of funding, with an amount much less than the typical funding for 
design and development grants whose focus is to test interventions designed to improve student 
learning. Few of the RPPs in this study had the resources to test new interventions at scale, nor 
were they set up to do so. Instead, they laid valuable foundations for future intervention research. 
These foundations are essential for intervention work to be grounded in a more thorough under- 
standing of the problem and to secure the mutual commitment of researchers and practitioners to 
pursue and test strategies for improving teaching and learning outcomes. 
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Perceived Benefits of Participating in a Partnership 


As a descriptive rather than evaluative study, we cannot make inferences about the impact of the 
partnerships in the RPP program. It is nevertheless useful to explore participants’ self-reported 
attitudes towards RPP work and their perceptions of the benefits of working in these partnerships. 
We share these findings in this section. 


Key Findings: 
e Participants highly valued working in their partnerships, with almost all survey 


participants agreeing or strongly agreeing that RPPs are worth the investment from IES, 
and that they would participate in another RPP in the future. 


e Inthe majority of partnerships, both practitioners and researchers were involved in 
collecting, organizing, and analyzing data as well as presenting at both research- and 
practice-oriented conferences. 


e About half of the partnerships had members who had written for traditional research 
outlets (i.e., articles, book chapters, or books). In addition, participants in half of the 
partnerships had contributed to new media platforms such as websites, blogs, webinars, or 
videos. 


e RPP members reported that partnerships provided a new framework or set of ideas and 
that they supported the design of PD or new programs or practices. 


e Participants from about one-third of partnerships reported contributing to a new or 
revised policy within the educational organization. 


Attitudes Towards Partnerships 

Consistent with patterns from our Phase I survey reports, both practitioner and researcher 
survey respondents in Phase II had very positive attitudes toward their partnership experiences. 
As shown in Figure 2, every participant (100%) agreed or strongly agreed that the opportunity 
to work collaboratively in an RPP was valuable to their work. Likewise, almost all respondents 
(96%-99%) agreed or strongly agreed that RPPs are worth the investment from IES, that they 
would participate in an RPP in the future, and that the research conducted by their RPP was 
meaningful to their own work. 


In addition, almost all respondents claimed that they trusted members of their partner organiza- 
tions (97%), and that members of their partner organizations trusted them (99%). Notably, these 
high self-reported levels of trust parallel the high level of reported progress on partnership goals 
related to building relationships as a foundation for partnership work, as presented in the “Prog- 
ress on Identified Goals” section above. 
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Figure 2. Participants’ attitudes toward working in researcher-practitioner partnerships, from survey 
(113 <n < 114 RPP researchers and practitioners). 


Contributing to Research and Dissemination Efforts 

RPP members have engaged in a variety of research activities, including data collection, data 
analysis, and dissemination of research findings to scholarly audiences, practitioner audiences, 
and the broader public (Table 5, below). In 22 partnerships, both researchers and practitioners 
reported presenting at conferences. They not only attended events hosted for their own groups 
(e.g., RPP researchers at the American Educational Research Association) but also conferences 
hosted for their partners’ communities (e.g., RPP researchers at the National Council for Teach- 
ers of Mathematics). In addition, nearly half of all survey respondents reported sharing research 
findings via multimedia platforms. As such, we suspect that the work of RPPs speaks to audiences 
within and beyond traditional research circles. 
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Table 5. Participation in Research and Dissemination Activities, by Role in Partnership 


Researchers Practitioners 

RPPs? Pls Others Co-Pls Others 
n 27 25 37 28 24 
Collecting, organizing, or analyzing data 24 5 36 25 18 
Presenting at a conference 22 24 30 18 13 
Contributing to a website, video, webinar, or blog 13 14 12 7 8 
Writing articles, book chapters, or books 12 17 23 8 7 


"Indicates the activity was endorsed by at least one practitioner in the partnership. 


Survey respondents also reported on the contributions of their partnerships’ work, particularly 
in terms of shaping policy and practice (Figure 3; see Appendix D for more). In 21 RPPs, at least 
one practitioner reported that their partnership provided a new framework or set of ideas to 
think about or address a focal problem. Partnerships were also involved in design work, includ- 
ing designing PD (19 RPPs) or integrating practices from the partnership into the routines of the 
educational agency (15 RPPs). 
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Figure 3. RPP contributions to local policy and practice, from survey (n = 27 partnerships). An RPP was 
included in each count if at least one practitioner from the partnership answered “yes.” 


Implications 

As in Phase I, we found in Phase II that there was a broad range of contributions made by RPPs 
that went beyond conducting and using research. In addition to traditional dissemination efforts 
(e.g., publishing scholarly articles), RPP members contributed to broader engagement strategies 
like presenting at conferences and sharing on multimedia platforms. Practitioners were heavily 
involved in conference presentations but were less involved in writing articles. Practitioners in a 
majority of RPPs reported that the partnership had offered new frameworks or ideas for thinking 
about their focal problem of practice and had contributed to design efforts for program imple- 
mentation as well. Given these findings, IES should continue to encourage a wide range of dis- 
semination, communication, and engagement activities in addition to traditional peer-reviewed 
journal articles. Likewise, RPP partners should be encouraged to attend or co-present at each 
other’s conferences. Specific encouragement and resources to include practitioners as co-present- 
ers might be valuable as well. 
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Research Use by RPP Practitioners 


In this section we turn to reports of research use among practitioners in the partnerships. 


Key Findings: 

e The RPP district leaders’ reports of research use look relatively similar to the patterns from a 
nationally representative sample of district and school leaders. Instrumental and conceptual 
uses of research were similar in both samples. Compared to the national sample, however, 
RPP district leaders reported less frequent symbolic use of research and more frequent reports 
of process use. 


¢ For all RPP practitioners, instrumental use of research was the most frequently reported type, 
followed by process, conceptual, and symbolic uses. Those in research roles were significantly 
more likely than those not in research roles to report higher levels of process use of research. 


e Among activities research evidence might inform, RPP practitioners were most likely to 
be involved in those related to directing resources to a program, scaling up a program, and 
designing PD. Although practitioners least frequently reported participating in purchasing an 
intervention or targeted program, they reported the highest frequency of research use for this 
activity. Instrumental use of research was also frequent in directing resources to a program or 
scaling up a pilot program. 

e Participation in key education decision-making activities varied by role group. RPP 
practitioners in non-research roles reported being more involved in activities related to 
purchasing an intervention or targeted program, redesigning a program, or designing PD than 
their colleagues in research roles. 


¢ RPP practitioners reported that the majority of their research partners were involved in 
central organizational decisions like directing resources to a program or designing PD. 


The Phase II survey asked how frequently RPP practitioners used research for different purposes. 
For the instrumental use scale, we first presented RPP practitioners with a list of activities 
common to educational decision making. We asked if their organization/district had engaged 

in each activity and, if so, whether they themselves had been involved in that activity and how 
frequently research had informed it. The items from the conceptual use scale asked participants 
how frequently they had encountered research that expanded their understanding of an issue, 
provided a new framework, or brought attention to a new issue. The symbolic use scale had items 
that asked practitioners to report how often they used research to convince others or to mobilize 
support. The process use items focused on when participants incorporated the processes of 
research into their own work—for instance, launching an evaluation study, participating in a 
grant proposal, or collaborating with others to analyze data. 


We first compared research use patterns for RPP leaders working in school districts (n = 31) 
against a nationally representative sample of school and district leaders (n = 733). We then 
considered patterns within the RPP sample (n = 51), attending to differences between RPP 
practitioners in research roles (m = 22) and non-research roles (n = 29). Finally, we examined who 
participated in key educational activities, looking at individuals in research roles versus non- 
research roles and involvement for IES RPPs in the past year. 
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Comparison Between Research Use of RPP District Leaders and a Nationally 
Representative Sample 

In 2015, NCRPP conducted a survey with a nationally representative sample of district and school 
leaders (n = 733). For this effort, we developed scales for instrumental, conceptual, symbolic, 
and process uses of research as described above. We used these same item banks for the surveys of 
RPP practitioners, allowing us to compare the different samples. 


Practitioners in IES RPPs worked in a variety of educational organizations (i.e., school districts, 
early childhood education organizations, departments of education and higher education, and 
social services agencies). Here, we compare only RPP participants who worked in school districts 
(n = 31) to the national sample. Although there were no school leaders in the RPP sample, 
school leaders’ reports of use did not differ significantly from those of central office leaders in the 
national sample, so we do not differentiate them here. 


As Figure 4 shows, instrumental use of research was the most frequent type of use among both 
RPP district leaders and respondents to the national survey, as compared to the other three uses 
of research. When we compare the two groups, the two biggest differences emerge for symbolic 
and process uses of research. First, RPP district leaders reported less frequent symbolic use 

of research (M = 1.97) than did the national sample (M = 2.49). Second, RPP district leaders 
reported more frequent process use (M = 2.53) then the national sample (M = 2.26). RPP district 
leaders reported slightly less frequent instrumental and conceptual use of research than the 
national sample. 


Instrumental Use 


Conceptual Use 


Symbolic Use 


Process Use 


1 


2 3 4 
mean ratings 


mw RPP District Leaders g National Sample 
(n = 31) (n = 733) 
Figure 4. Practitioners’ instrumental, conceptual, symbolic, and process use of research, from RPP survey 
and national survey (n = 31 RPP district leaders; n= 733 district and school leaders in national sample). 
1 = never; 2 = sometimes; 3 = frequently; 4 = all of the time. 
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RPP Practitioners' Use of Research, by Research Role 

When we return to the broader sample of RPP participants, we see that almost half (43%) of the 
51 practitioners who participated in Phase II were in research-related roles in their organizations. 
Since practitioners’ use of research can differ based on their roles,** we wondered if the RPP 
participant patterns would look different when we considered whether they were in researcher 
versus non-researcher roles. 


To address this question, we disaggregated the reports of research use for RPP practitioners who 
were in research-related roles (n = 22) and those who were not in research-related roles (n = 29). 
Figure 5 shows the mean scale scores for all four types of use in Phase II for all practitioners, as 
well as by research role. 
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Figure 5. Practitioners’ mean reported use of research, by type of use and role in organization, from Phase 
Il survey. For instrumental use, only those who reported involvement in an instrumental use activity are 
included (20 < n < 32 RPP practitioners). For all other use reports, n = 51 RPP practitioners. 

1 = never; 2 = sometimes; 3 = frequently; 4 = all of the time. 
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RPP Practitioners’ Participation in Activities and Instrumental Uses of Research 
Next, we looked more closely at reports of instrumental use among RPP practitioners. 


Overall, across all RPP practitioners (n = 51), involvement in instrumental use activities ranged 
from a high of almost two-thirds of practitioners (31 to 32, or 61% to 63%) involved in directing 
resources to a program, scaling up a program, and designing PD, to a low of just under two- 
fifths (20, or 39%) participating in purchasing an intervention/targeted program or redesigning 

a program (Table 6). The majority of practitioners who were engaged in these decision-making 
activities reported using research “frequently” or “all of the time” for all of them. Notably, 
although only 39% of practitioners said they were involved in purchasing an intervention or 
targeted program, they reported the highest frequency of research use for this category (91% said 
“frequently” or “all of the time”), which is consistent with federal policy guidelines that require 
the use of research evidence in selecting and purchasing interventions. 


Table 6. Percentage of Participating Practitioners and Co-Pls Reporting Involvement in 
Educational Decisions 


For those who indicated that they were involved 
in an activity, how frequently was researched used? 


# of RPP 
Practitioners 
engaged in Never Sometimes Frequently All of the time 

activity 

(n= 51) 
Directing resources to 63% 0% 26% 55% 19% 
program 
Scaling up pilot program 61% 0% 32% 42% 26% 
Designing professional 61% 3% 33% 40% 23% 
development 
Eliminating program or policy 47% 4% 30% 48% 17% 
Purchasing intervention or 39% 0% 11% 58% 32% 
targeted program 
Redesigning program 39% 0% 40% 40% 20% 


In our Phase I analysis, we noticed less involvement by those in research roles in the key activities 
of the instrumental use items as compared to all RPP practitioners. We wondered if this pattern 
would be replicated in our Phase II survey reports. We hypothesized that participants in research 
roles within a practice organization might not always be centrally involved in key decisions within 
the organization, and that these decisions might be within the purview of other leaders. 


Consistent with this hypothesis, we saw in Phase II that RPP practitioners in different roles were 
not equally involved across all activities (Figure 6; see Appendix E for more). Practitioners in re- 
search roles were less likely than their colleagues in non-research roles to be involved in activities 
related to purchasing an intervention or targeted program (32% vs. 45%, respectively), redesign- 
ing a program (27% vs. 48%), or designing PD (45% vs. 72%). There were only slight differences 
between these two groups in reported participation in the other activities listed. 
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Figure 6. Practitioners’ involvement in educational decision-making activities, by role in organization, from 
Phase II survey (n = 51 RPP practitioners [29 RPP practitioners in non-research roles and 22 RPP practi- 
tioners in research roles)]). 


RPP Involvement in Decision-Making Activities 

Finally, we analyzed whether the research partners in the RPP had been involved in key activities 
for educational decision making during Phases I and II. In both phases of the survey, RPP 
practitioners who indicated that they had been involved in the decision-making activities listed 
above were further asked whether their IES RPP partners had been “consulted or involved” in the 
activity. Such involvement would be evidence of a different, more direct form of use of research to 
support decision making, and we found such evidence in many of the RPPs. 
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Table 7. Involvement of RPPs in Central Organizational Decisions 


# of RPPs with at Least One # (%) of RPPs with Research 


Practitioner Involved in Activity Partner Involved in 
in Phase I or II (n = 27) Phase I or II 
Directing resources to program 25 23 
(92%) 
Designing professional development 24 20 
(83%) 
Scaling up a pilot program 24 19 
(79%) 
Eliminating program or policy 24 19 
(75%) 
Purchasing intervention or targeted 19 16 
program (84%) 
Redesigning program 21 14 
(67%) 


Practitioners in a majority of RPPs reported that their research partners had been involved in 

all of the common decision-making activities listed here. Practitioners who had been involved 
reported that their research partners were most frequently involved in directing resources to 

a program (92%, or 23 of the 25 RPPs where at least one practitioner had been involved in the 
activity in their organization). Around three-quarters of RPP partners were reported to be in- 
volved in scaling up a pilot program (79%) or designing PD (75%) when that activity occurred in 
a practice organization. Over half of RPP partners were reported to be involved in redesigning a 
program (67%) or purchasing an intervention or targeted program (84%) when practice organiza- 
tions engaged in that activity. 


Implications 

When policymakers encourage district leaders to use research to inform their decision making, 
there is either an explicit or implicit theory of action that posits research findings can and 

should directly shape decisions related to policy or practice, particularly related to program or 
intervention adoption. Findings here suggest that the purposes for which RPP practitioners 
reported using research were broader than just instrumental use, which is consistent with findings 
from our nationally representative survey and past research on evidence use. We found evidence 
not only of leaders’ instrumental use of research emphasized by policies but also of conceptual, 
symbolic, and process uses of research. 


Looking within the RPP participants, we see an interesting pattern related to practitioners and 
whether they are in research or non-research roles. On one hand, individuals in research roles 
may be well positioned to engage in process use of research, perhaps by designing and helping 
conduct an evaluation study or by analyzing data with others within their organization. On the 
other hand, these individuals may not be as likely to be involved in key educational decisions such 
as those related to purchasing programs, re-designing programs, or designing PD. Future research 
could identify the alternative organizational decisions that involve individuals in research roles or 
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perhaps consider how those in research roles can support research use with colleagues in other 
roles. 


Finally, RPP research partners were involved in some central decisions such as designing PD, 
as reported here and in other sections. Given that these are key decision areas for educational 
agencies, this is a significant contribution for RPPs. 
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Research That RPP Practitioners Found Useful 


In this section we report on an open-ended item included on the Phase I survey that asked 
practitioners to name a piece of research that “changed your thinking or opinions about possible 
solutions to your organization's/district’s problems.” We located the sources named and coded 
each one for format (e.g., book, journal article, research/policy report), topic, disciplinary content 
area, student subgroup focus, and the reason that the practitioner found the research useful. RPP 
practitioners’ responses are compared to responses on the nationally representative survey. 


Key Findings: 

e RPP practitioners most often named journal articles as the pieces of research they found 
most useful in the past year, and the research most often focused on student learning or 
school organization. National survey respondents typically named books, and the work 
most often focused on teachers and instruction. 


e Most of the research named by RPP practitioners as well as national survey respondents 
did not focus on particular disciplinary content areas. 


e RPP practitioners named research focused on student subgroups more frequently than 
national survey respondents did. 


e The reasons that RPP practitioners found their research sources useful typically related 
to designing programs, policies, and initiatives; national survey respondents frequently 
named reasons related to supporting their own professional learning. 


Of the 43 practitioners who responded to the Phase I survey, 26 (60%) completed this item with 
enough information to identify the source. These 26 respondents represented 17 of the 25 RPPs 
(68%) for which at least one practitioner responded to the survey.*” Similarly, of the 733 school 
and district leaders who responded to the national survey, 377 (51%) completed the open-ended 
item. Comparisons of district leaders in the RPP sample to those in the national sample, as well 
as comparisons of those in research roles in both samples, resulted in negligible differences from 
the patterns found by comparing the full RPP and national samples. Therefore, we report the 
patterns below for all RPP and national sample respondents. (See Appendices G-I for additional 
information.) 


Format of Research Named 

Figure 7 compares the formats of the pieces of research named by RPP practitioners and national 
survey respondents (see Appendix F for more). Overall, RPP practitioners most frequently named 
journal articles (46%), whereas national survey respondents most often named books (58%). In 
fact, a striking 84% of RPP practitioners who answered this question named either journal articles 
or research/policy reports, compared to only 31% of national survey respondents naming either 
one of these two formats. 


Technical Report No. 3 54 
National Center for Research in Policy and Practice 


46% 
Journal article 


ee 


14% 


38% 
Research/policy report 


Book 


58% 


Other 
12% 


0% 10% 20% 30% 40% 50% 60% 70% 
% of practitioners 


mw RPP district leaders (n=26) National sample (n=377) 


Figure 7. Format of research named as useful by RPP practitioner survey respondents (n = 26 RPP practi- 
tioners) and national survey respondents (n = 377 district and school leaders). 


Topics of Research 

Figure 8 presents the most frequent topics of the pieces of research named by RPP participants 
and national survey respondents. (See Appendix G for more information on topics and 
subtopics.) RPP study respondents most frequently named sources focused on students and 
learning (33%) or school organization and improvement (31%). In contrast, national survey 
respondents most frequently named pieces focused on teachers and instruction (36%), especially 
pedagogical strategies. On the national survey, research that focused on students and learning 
(28%) was most often related to academic achievement. 


We compared the topics named by RPP practitioners to the focus areas of their RPPs, as they 
responded to the survey in the context of being RPP leaders. Indeed, RPP practitioners often 
named pieces of research connected to the focus areas of their RPPs. For example, all of the 

RPP practitioners who named research on the subtopic of postsecondary readiness and learning 
(including K-12 leaders) represented RPPs focused on higher education or adult learning. 
Likewise, RPP practitioners named research related to school-level interventions more frequently 
than did national survey respondents, which tended to reflect the responses of practitioners 
whose RPPs focused on identifying or implementing interventions to support kindergarten 
readiness or K-12 students with special needs. 
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Figure 8. Topics of research named as useful, from RPP survey and national survey (n = 36 topics named by 


RPP practitioners; n = 552 topics named by national sample). The n may exceed the number of respondents 
because some pieces of research addressed multiple topics. 


36% 


Disciplinary Content Areas of Research 

For both RPP study respondents and national survey respondents, 81% of the research that 
was named did not focus on a particular disciplinary content area. The research that did focus 
on a content area reflected either literacy (8% of RPP survey responses; 10% of national survey 
responses) or STEM-related topics (11% of RPP survey responses; 9% of national survey 
responses). 


Student Subgroups in Research 

As shown in Figure 9, when naming research that was useful to them, RPP practitioners more 
frequently named research that focused on different specific student subgroups (58%) than did 
national survey respondents (31%; for more, see Appendix H). This difference was especially 
related to a larger proportion of RPP practitioners naming research that focused on education for 
students with special needs, such as tiered interventions (23%), which was often named by those 
in partnerships focused on special education. The higher proportion of RPP practitioners naming 
research focused on students’ socioeconomic status was not as directly related to RPP focus areas 
but may have informed RPP goals related to school improvement and academic outcomes. 
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Figure 9. Subgroups reflected in pieces of research named as useful, from RPP survey and national survey 
(n = 26 subgroups named by RPP practitioners; n = 395 subgroups named by national sample). The n may 
exceed the number of respondents because some pieces of research were coded for multiple subgroups. 


Reasons Leaders Found Research Useful 


For each piece of research a respondent named, we also asked them to respond to this prompt: 
“Please describe how this piece of research was useful to your work.” Similar to patterns of 
responses on the national survey, 81% (21 of 26) of RPP practitioners who responded to the full 


open-ended item gave enough information for our team to code. 


As shown in Figure 10, of those responding to this part of the item, RPP study respondents more 
frequently named reasons for usefulness that were related to designing programs, policies, and 

initiatives (48%). National survey respondents, on the other hand, most frequently named reasons 
related to supporting leaders’ own professional learning (29%), closely followed by designing 


policies (27%) and providing instructional leadership (25%). 
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Figure 10. Reasons that practitioners found a piece of research useful, from RPP survey and national survey 
(n = 21 RPP practitioners; n = 286 district and school leaders from national sample). 


Implications 

This analysis provides insight into the types of evidence that practitioners have in mind when 
they claim to use research for a variety of purposes. Across both samples, we see a broad range of 
sources for research that goes beyond traditional journal articles. 


It is noteworthy that RPP study practitioners answered this item differently than did respondents 
in the nationally representative sample. One striking trend is that RPP practitioners most often 
named journal articles, whereas national survey respondents most often named books. Further, 
for almost three-quarters (73%) of RPP practitioners, the topic of the research matched the 

focal topic of their RPP. It is possible that the research they cited aligned to the focus of their 
RPPs’ work because they were asked to participate in the survey as RPP leaders. Alternatively, 

it could speak to the type of research that is read by educational leaders who choose to work in 
RPPs focused on those problems of practice. Or, it may be that the RPP created opportunities for 
practitioners to engage with peer-reviewed journal articles around those topic areas. Without an 
experimental design and with such a small RPP sample (n = 26), however, it is important not to 
attribute these differences to the efforts of RPPs. 


In order to gauge how RPP practitioners came to use research related to their RPPs’ focal areas, 
future survey items might ask practitioners to specify where they first learned of and accessed the 
research. Of course, such items are subject to problems of remembering, but it would be valuable 
to more precisely trace where research is being accessed. Given the IES RPP program's goal of 
influencing practitioners’ research use, initial and exit surveys that include this and other key 
aims could be worthwhile. 


Technical Report No. 3 58 
National Center for Research in Policy and Practice 


Reported Shifts in Partners' Engagement with Research 


RPPs may create conditions that support shifts in how practitioners and researchers engage with 
or use research in their work. Below, we report on self-reported shifts in both groups’ practices. 


Key Findings: 
e The majority of surveyed RPP practitioners said that they had become better at using 
research in their work and were more likely to do so because of the partnership. Likewise, 


almost all of surveyed RPP researchers agreed that they had become better at conducting 
research that meets the needs of practitioners. 


¢ Both RPP practitioners and researchers reported that they had gained new skills for 
working in partnerships and for conducting research relevant to their organizations’ needs 
and contexts. 


e A majority of RPP practitioners described gaining an appreciation for the value of 
research and becoming more open to participating in and using research in the future. 
Likewise, a majority of researchers reported gaining an appreciation for practitioners’ 
contexts and the value of their input in the research process. 


e Practitioners and researchers both discussed new skills they had developed in 
communicating research to a variety of stakeholders. 


e Almost all of those surveyed from both groups agreed they would be confident leading a 
future partnership. 


The Phase II survey asked practitioners (n = 52) and researchers (n = 62) about the extent to 
which their own skills in using research and leading an RPP had shifted due to working in their 
partnership. Likewise, in both Phase I and Phase II interviews, we asked practitioners, “How, if at 
all, has your own use of research changed since you began participating in this partnership?” We 
asked researchers, “How, if at all, have you seen changes in the extent to which your partners use 
research?” and “Have you noticed any changes in the way that you've done research as a result of 
working in this partnership?” 


We posed these questions to a total of 54 practitioners and 70 researchers in one or both phases 
of the study. In order to capture patterns of reported shifts, we counted the occurrence of each 
theme for individual participants, whether that theme was mentioned by that participant in one 
or both phases. For example, if an interviewee participated in Phase I and Phase II of the study 
and mentioned the same theme in both interviews, that theme was counted only once. On the 
other hand, if the interviewee mentioned different themes in each phase, we attributed each 
theme to that individual's reports. 


In the pages that follow, we first describe survey and interview findings related to shifts in 
engagement with research among practitioners, followed by shifts among researchers. 
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Shifts in Practitioners' Engagement with Research and Partnership Skills 

Survey findings. The Phase II survey for RPP practitioners included four items about changes in 
research use and partnership skills. As shown in Figure 11, over three-quarters (77%) of surveyed 
practitioners (n = 52) agreed or strongly agreed that they had become better at using research to 
inform their work because of the partnership, while over two-thirds (68%) said they were more 
likely to use research in their work. 


Further, practitioners reported positive shifts in their abilities to engage in or lead a partnership. 
Almost all (90%) agreed or strongly agreed that they felt confident they could lead a partnership 
in the future, while four-fifths (81%) reported they had developed new skills in working in a 
partnership through this project. 
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Figure 11. Practitioners’ reports of shifts in engagement with research due to working in RPPs, from Phase 
Il survey (51 <n < 52 RPP practitioners). 


Interview findings. Over three-quarters (79%) of the 54 practitioners and over two-thirds (69%) 
of the 70 researchers who were interviewed reported changes in how practitioners and others in 
their practice organizations engaged with research. For example, some practitioners noted that 
their colleagues were more receptive to participating in and using research because their experi- 
ence with the partnership's study had been relevant and useful for their work. 


Among the remaining 21% of practitioners and 31% of researchers who claimed that practi- 
tioners’ use of research had not changed, participants frequently explained that they were already 
deeply engaged in using research through their roles in their organization (i.e., they were in 
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research-related roles), and/or they had worked in partnerships before the IES RPP grant, such 
that they could not attribute any changes solely to this particular project. Still others pointed out 
that it was too short of a time to see a notable shift. 


As shown in Figure 12, participants described three primary types of changes in RPP 
practitioners’ engagement with research. Practitioners most often described a change in their 
orientation toward using and participating in research (57% of the 54 interviewed), and this was 
the second most frequently named shift among researchers (40% of the 70 interviewed). About 
half of practitioners (49%) and researchers (56%) described changes related to practitioners’ 
development of knowledge and skills to use and conduct research. And a minority of both 
practitioners (26%) and researchers (10%) explained how practitioners communication with 
stakeholders about research had shifted due to working in the partnership. We describe each of 
these themes in more detail below. 


Orientation to research 


Knowledge and skill building 
about the research process 


Communication with 
stakeholders 


No change 
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Figure 12. Practitioners’ and researchers’ reports in Phase | and Phase II interviews of shifts in how practi- 
tioners use research due to working in RPPs (n = 124 RPP participants [54 practitioners, 70 researchers)). 
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Practitioners shifted their orientation to research. Participants who described changes in prac- 
titioners’ orientation to research frequently noted an improved understanding of the value of 
research to shed light on problems of practice and make evidence-informed decisions in their or- 
ganization. Some practitioners expressed that both they and their colleagues had gained an appre- 
ciation of how research could improve their work and had come to view research and researchers 
as a reliable source of information. One practitioner shared, for example, “For me, it definitely has 
reinforced the value of research, because we wouldn't be aware of the successes or the challenges 
or even the capacity to roll something out to a scale like this without the information from our 
partnership.” 


Other practitioners added that their increased awareness about the value of research had “elevat- 
ed the importance of accurate data for us and for our local programs” in the organization more 
broadly. In this way, developing an understanding about the value of research affected not only 
practitioners who participated in the RPP but their colleagues as well. One practitioner explained 
this shift in her colleagues’ use of research in decision making: 


Because of their exposure [to research from the partnership], they are more 
open, and it’s natural now to say, “Well, what does the research say?” For them 
to [be] thoughtful about doing research on their own, and actually being open 
to using research for instructional methods or for curriculum design. 


Another practitioner noted a shift in colleagues’ openness to participating in research and work- 
ing with external researchers due to their positive experience in working in the IES RPP: 


We now seek out the opportunities to work with others in the same kind of 
capacity. In my [research and evaluation] office, we have the people that could 
do the work, but timewise, we just can’t. There's not the time to really dig into 
these kinds of things, so leveraging researchers and others to come in as part of 
that kind of mutual benefit thing .... [W]e’re more open to it or seek it out a little 
bit more than maybe we did in the past. 


Researchers likewise noted shifts in practitioners’ attitudes toward using and participating in 
research. A researcher in one partnership noted that his practitioner partners have “definitely 
become more open to research.” He elaborated: 


They basically said, because of the success of this RPP, they really didn’t need 

to be persuaded to work with [researchers] or to take on research. Folks in the 
district office have been really impressed by the kinds of information that’s been 
coming out this work, and they’re actually actively looking for other research 
projects because of that. So that was really encouraging to hear. 


Practitioners developed knowledge and skills about the research process. Practitioners and research- 
ers alike noted shifts that resulted from practitioners conducting research as part of their partici- 
pation in the RPP. Through participation in the research process and sustained engagement with 
both qualitative and quantitative data, practitioners reported gaining a better understanding 
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of their focal problems of practice. 


As part of this process, practitioners gained new technical skills related to collecting, managing, 
analyzing, and interpreting data. Some partnerships hosted data management workshops and 
other forms of support that helped practitioners gain technical skills to conduct research in their 
own organizations. One practitioner emphasized the importance of this shift in his work: 


One of the huge benefits of the researcher—practitioner partnership is they 

[the research team] were so willing to do one-on-one tutorials with [us] to 

build our analytic capacity. ... Now, we can do it on other surveys that we 
administer, and ... share [our] expertise with other districts who are interested 

in replicating some of the analyses we've done. ... Because they built so much 
analytic capacity on our side, we were able to translate, understand, and interpret 
the research much better so that we can think of more creative and user-friendly 
ways to display that data. 


As practitioners gained research experience, they became more critical consumers of research. 
Researchers especially noticed that practitioners began to better evaluate the quality of research 
studies and validity of findings. One researcher observed that instead of turning to “Dr. Google,” 
her practitioner partners began to find more reliable sources of evidence, including researchers, to 
inform their practice. A practitioner from a different RPP made a similar observation: 


I'm less likely to just go with the headline on something. I want to know more, 

I want to go deeper, and I want to make sure that the direction that we're going in 

is based on research, it’s got a deeper foundation. It’s not what's the latest sales pitch. 
... ’m more thoughtful in my purchases. ’m more thoughtful in the practices that 
we adopt. ... I really think that we’re making a difference, and I know I couldn't 
have done it without the partnership. Just having access to world-class researchers 
and the guidance, even though they didn't tell us one thing to do. Just the thoughtful 
questioning and looking at what we were doing—I really think that’s been big. 


Researchers further explained that it was important that practitioners developed understandings 
about the generalizability of findings and became aware of the limitations of research. Similarly, 
with increased knowledge and skills about the research process, practitioners and researchers 
alike noticed how practitioners asked new questions about the partnership's study and about their 
own focal problems of practice. One practitioner reflected that this new experience gave him a 
sense of ownership in the research: 


I was seeing, well—What are we looking at? Why are we doing this? What's 
the purpose? ... Then all [of a] sudden, it becomes very personal because then 
you have a vested interest in it. My going out and gathering the data gave me 
that vested interest in it. You have buy-in. You have ownership. ... I think that 
would be the biggest shift. People are questioning. 
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Taken together, these shifts suggest increased sophistication in practitioners’ understandings 
of developing and implementing research studies, as well as of interpreting and implementing 
research findings. 


Practitioners shifted their communication practices with stakeholders. 
Practitioners noted that the partnership helped them gain the skills necessary 
to communicate complex research findings to various stakeholders. In doing so, 
they worked to strike a balance between what they called “technical” and 
“friendly” communication. Similarly, practitioners described efforts to 

balance their production of publications and presentations across venues 

that met the needs of various stakeholders. For example, one practitioner 
mentioned creating a short “two-pager” to summarize their work, while 
another described a video created with participants from the partnership’s 
study, which was posted to the district’s website. 


One practitioner explained both the importance of and effort involved in communicating re- 
search well: 


[It] has been a tremendous learning curve. I’ve had to develop more skills 

in that regard than what I’ve had to use in the past. My experience has 

afforded me to be a messenger or an interlocutor to those other communities, 
like superintendents of schools, legislature, and others. Having been involved in 
this research community has made that a whole lot easier for me and for 

them relative to their own sense of how research serves their interest. 


Practitioners’ leadership in the dissemination of findings to a wide variety of audiences—in- 
cluding administrators, educators, students, families, policymakers, legislators, researchers, and 
more—requires skills in both interpreting and communicating research findings through a vari- 
ety of formats and styles. 


Shifts in Researchers' Approach to Research and Partnership Skills 

The reported shifts in practices or skills were not one-sided and limited to practitioners. Indeed, 
the researchers involved in the RPPs also reported notable changes in their approaches to research 
and the development of a broader set of partnership skills. 


Survey findings. The Phase II survey for RPP researchers (n = 52) included three items about 
changes in researchers’ research and partnership skills. As shown in Figure 13, the great majority 
(88%) of researcher respondents agreed or strongly agreed that they had become better at 
conducting research that meets the needs of practitioners. As in practitioners’ reports described 
above, almost all researchers reported feeling confident that they could serve as a leader in a 
future partnership (95%) and that they had developed new skills from working in a partnership 
(94%). 
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Figure 13. Researchers’ reports of shifts in approach to research due to working in RPPs, from 
Phase II survey (n = 62 RPP researchers). 


Interview findings. Of the 70 researchers who participated in an interview in one or both phases of 
the study, 55 (79%) described ways in which their approach to conducting research had changed 
due to working in the partnership. The remaining 15 (21%) said that they had not noticed much 
change in their research. Almost all attributed this to having worked in partnerships for a long 
time or being predisposed to doing research in this manner. 


Figure 14 shows the three primary themes regarding changes to researchers that emerged from 
our interviews. Parallel to participants’ descriptions of the changes they noticed in practitioners, 
researchers discussed shifts in their orientation to working with practitioners (54%); their devel- 
opment of knowledge and skills related to doing research with practitioners (41%); and changes 
in their communication with stakeholders about research (36%). In addition, although we did 
not ask practitioners directly about changes to researchers’ approaches, about one-quarter (28%) 
of them mentioned changes along the same lines that they had noticed in their research partners 
during the course of their interviews. We describe each of these themes in more detail below. 
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Figure 14. Researchers’ reports of shifts in how they do research due to working in RPPs, from Phase | and 
Phase II interviews (n = 70 RPP researchers). 


Researchers shifted their orientation to working with practitioners. Researchers who noticed 
changes often mentioned ways in which their orientation toward working with practitioners 
had shifted. Partnerships created opportunities for researchers to engage in educational settings, 
contributing to an improved understanding of practitioners’ contexts. Here, researchers described 
learning about both the policy contexts and organizational conditions that shape practitioners’ 
work. They often described this deepened understanding as a precursor to adapting research aims 
and methods to accommodate practitioners’ needs and to improving their ability to interpret data 
and produce actionable findings for both practitioners and policymakers. As one researcher put 
it: 


I’ve become more aware of the policy context and the political context. 
I think more critically about what role research can play in informing 
public debates, and how people access research, how to set up research 
studies, so that they result in actionable findings that policymakers and 
practitioners can use. 
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Some practitioners echoed the importance of researchers gaining awareness of the policy climate 
in their partnerships’ areas of focus, making observations such as, “The research team has really 
learned a lot about how early childhood policy is being set in our state and all the politics around 
it; 


Further, researchers often discussed shifts in how they considered practitioners’ input in deter- 
mining the aims and central questions of a research project. They described this as a shift away 
from what they called “traditional” approaches of entering an educational setting to conduct a 
study without practitioner input. Instead, their approach to research in the partnership involved 
asking questions and listening to practitioners’ perspectives and expertise in order to align re- 
searchers and practitioners’ goals: 


The other thing that I learned from this ... is that what we think is needed 
based on where we sit ... may not actually reflect the reality in the schools, 
even if it’s our job to know what they need. It’s important to get into the 
buildings and walk around and listen and ask questions. ... I've learned 
a lot from them on how you approach things, how you be respectful to 
people and listen to them and work with them in a way where you can 
continue to work with—rather than parachuting in as some sort of 
national expert on something on your way to the next location. 


Researchers developed knowledge and skills about conducting research with practitioners. An- 
other key shift that researchers described in their work related to changes to their research meth- 
ods to better align with practitioners’ needs and contexts. For example, one researcher claimed 
that he previously focused on randomized controlled trials but realized that to answer the part- 
nership’s focal questions within a school district context, they needed to pursue a mixed-methods 
study. Another researcher experienced a similar shift: 


One thing that has changed for me as part of being in this partnership ... 
is the value that I see in qualitative approaches. I’ve quantitatively trained, 
but I really am starting to think [that] practitioners really need to 
understand the mechanisms underlying why things are happening. 


In this way, researchers talked about becoming more flexible in deciding on the most appropriate 
research methods to use. Some researchers also noted that they developed new technical skills as 
they worked with new data systems in educational organizations and learned how to create and 
work with data sharing agreements. 


Just as researchers developed flexibility in the research process, they learned how to develop more 
timely and relevant findings with their practitioner partners. As one researcher claimed, “We're 
much more reactive to the current needs of people here, in terms of what they need, so we're [do- 
ing] much faster-paced, simpler analyses.” Some researchers noted that this shift was challenging, 
as they had to become comfortable with “quick and dirty” analyses that required them to carefully 
communicate limitations and nuances to stakeholders. 
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Sharing findings, however, did not signal the end of the partnership's collaboration. Several 
researchers noted the importance of continuing to work with their partners in planning for the 
implementation of research findings. As one researcher explained, “It’s an entirely different frame 
than most research that I’ve done. I’ve always done research in the schools. But I think, most of 
the time, we have our research questions, and we go in, and we answer them, and then we leave.” 
Instead, this researcher explained, their partnership's collaborative research process began with 
the initial joint development of research questions and aims, continued through the study itself, 
and persisted through the application of findings to the design of policies and programs. 


Researchers shifted their communication practices with stakeholders. As with practitioners above, 
researchers talked about changes in communication with stakeholders as a result of working in 
their partnerships. They described how they had learned to communicate research findings in an 
accessible and understandable way for educators. This involved not only shifting from academic 
to “friendlier” language but also constructively interpreting and communicating findings in ways 
that were both respectful of practitioners’ contexts and actionable for their needs. As one re- 
searcher explained, doing so sometimes meant involving practitioner partners in communication 
strategies: 


I think we ... are constantly growing and learning how to be better partners 

in terms of writing, communicating our findings in ways that are appropriate 
for practitioners and policymakers—not writing journal articles or long reports 
[but] communicating our findings in a variety of more acceptable ways. We're 
always trying to negotiate and figure out what's the right time to get them 
involved in different aspects of either the research or sharing of the findings. 


In addition to ensuring clear and accessible communication, researchers emphasized the impor- 
tance of stating both possibilities and limitations related to the findings so that they would not be 
taken out of context or used to make erroneous inferences. Clarification about the interpretation 
of findings was especially important because both researchers and practitioners communicated 
their findings to multiple stakeholders in various settings including conferences, presentations, 
and trainings. 


Implications 

In survey and interview data alike, we found evidence that both practitioners’ and researchers’ 
experiences in their partnerships may have contributed to an increase in practitioners’ capacity 
to engage with research and researchers’ capacity to conduct research that is relevant to 
practitioners’ needs. Many of the researchers and practitioners involved in the IES RPPs shared 
very positive reports on the ways in which the RPP had contributed to their work and knowledge 
base. While these themes are all based on self-reports, the findings are notable. 


In both the survey and interview reports, most practitioners said that they had become better at 
using research in their work and were more likely to do so because of their participation in the 
partnership. They described shifts in their own and their colleagues’ orientation toward research 
in terms of their understanding of its value and their openness to participate in studies and use 
the findings. In addition, their experience in conducting research in their partnerships expanded 
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their knowledge and skills about how to ask research questions, choose and implement appropri- 
ate research methods, and interpret and communicate findings. These perceived shifts in practi- 
tioners’ engagement with research suggest that their involvement in RPPs can influence not only 
their own research use, but also use within their organization more broadly. 


Similarly, many of the RPP researchers reported that they had become better at conducting re- 
search that meets the needs of practitioners. Their orientation toward working with practitioners 
likewise shifted as they learned about practitioners’ contexts, included their input in the research 
process, and developed collaborative relationships with them. As researchers adapted to practi- 
tioners’ needs and contexts, they became more flexible in their research methods, in producing 
timely and relevant reports of findings, and in communicating findings in an accessible way for a 
variety of stakeholders. 


It is important to consider that there were subsets of both researchers and practitioners who, in 
their interviews, shared that they had not shifted their practices or knowledge base in notable 
ways. These individuals included practitioners who were in research roles in their respective 
educational organizations, many of whom had traditional research preparation through doctor- 

al programs and felt confident in their research or partnership skills. This group also included 
researchers who had long histories of conducting collaborative research with practitioners. In 
other words, these participants’ experiences in the RPP were “business as usual.” In such cases, the 
IES RPP grant supported their work together, but did not, in their eyes, help them to bring en- 
tirely new skills to bear on it. This suggests that the IES RPP program might consider the range of 
experiences of the researchers and practitioners involved in each RPP. The program might benefit 
from including researchers with prior experience in partnership work as well as practitioners with 
experience in research in order to provide a foundation for a successful partnership, while also 
including members with less experience in these areas in order to offer new experiences that build 
capacity. 
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Nature of Prior Relationships of the RPPs 


In our Phase I interim report, we reported that there were few truly new RPPs funded through the 
IES grant program—that is, partnerships where individuals or organizations had never worked 
together before. In Phase II, we obtained more specific information about how the partners had 
worked together prior to the grant. 


Key Findings: 

e The majority of partnerships had participants who worked together before receiving IES 
RPP funding, and most participants knew at least one person in their partner organization 
before the grant started. 


e The majority of participants had worked on at least one project with their RPP partner for 
several years prior to securing IES funding. 


¢ Before the IES grant began, 19 partnerships had established a formal data-sharing 
agreement, 18 had established a broader research agenda beyond the focus of the IES 
grant, 16 had established an MOU, and 11 had established a decision-making board. 


In our Phase II survey, participants from all 27 partnerships indicated that members of 
organizations in the partnership had worked together in some capacity prior to securing IES RPP 
funding. Eighty-seven percent of survey respondents, including 22 of 25 PIs and 26 of 28 co-Pls, 
noted that they or members of their organization had worked with their partners before the IES 
RPP grant began (see Appendix A for more). 


We asked how long the partners had worked together before the IES grant began (see Figure 15). 
Slightly over half (51%) of respondents reported that they had worked with their RPP partners 
between one and three years prior to receiving the IES grant; over one-third (36%) reported 

that they had worked together more than three years before receiving funding. The remaining 
individuals (13%) reported working together less than a year before the IES grant. 
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Figure 15. Length of time RPP partners had worked together before IES Grant, from Phase II survey 
(n = 97 RPP researchers and practitioners). 


We asked survey respondents whether their partnership had any formalized structures in place 
prior to securing IES funding. Most of the RPPs (26 of the 27) had developed at least one of these 
pieces of infrastructure prior to securing the grant funding. As seen in Figure 16, 19 RPPs had es- 
tablished a formal data sharing agreement, 18 had established a broader research agenda beyond 
the focus of the IES grant, 16 had established an MOU, and 11 had established a decision-making 
board prior to receiving the IES grant. Only one RPP indicated their team had worked together 
prior to receiving IES funding without engaging in any of these four types of formalized partner- 
ship activities. 
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Figure 16. Number of RPPs with key structures in place before IES grant, from Phase II survey (n = 27 RPPs). 


Implications 

Together, these findings support our previous conclusion that the RPP program’s RFA and review 
process has attracted partnerships with a history of working together. While the RFA notes 

that the program supports both new and established partnerships, we see a tendency toward 
partnerships with prior relationships and structures to support partnership work—such as 
MOUs—treceiving funding. We do not yet know, however, how funded proposals compare to non- 
funded proposals with respect to prior collaborations. As such, the program may want to consider 
the following questions: 


Are partnerships with a prior history more likely to be funded? If so, does 
that suggest new guidance for the program regarding the potential benefit 
of applicants first developing a relationship through a small project before applying? 


Might there be different expectations for goals or timelines for newer partnerships 
compared to well-established partnerships that already have infrastructure in place, 
including data-sharing agreements, MOUs, or research agendas? 
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Conditions for Launching and Maintaining Partnerships 


In Phase I interviews, we asked participants to describe the conditions that had supported the 
development or maintenance of their partnerships. We used their open-ended responses to create 
a list of the most-frequently named conditions and then included these in our Phase II survey. 


Key Findings: 


e ‘The top two conditions for launching an RPP were mutual organizational interest and 
trust among RPP members. Other conditions that supported starting an RPP included 
a data-sharing agreement or MOU, the individual expertise of RPP members, and 
organizational leadership. 


¢ Holding regular meetings, having mutual organizational interest, and establishing trust 
among RPP members were top conditions for maintaining a partnership. 


We asked survey respondents to indicate the top three conditions that supported the launch 

of their RPPs (Figure 17). Participants across role groups most often noted that having mutual 
organizational interest in a particular focus area was an important condition for launching a 
partnership (29% of respondents). Having trust among RPP members was the second most 
named condition (15%), followed by the individual expertise of RPP members or the presence of 
a data-sharing agreement or MOU (11% each; see Appendix B for more). 
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Figure 17. Most important conditions for launching an RPP, from Phase II survey. Respondents selected up 
to three conditions from a list of 10 options (n = 332 selections). 


In terms of maintaining the partnership, 25% of survey respondents noted the importance of 
holding regular meetings (Figure 18; see the interim report for additional information on fre- 
quency and type of meetings). Respondents agreed that having mutual organizational interest in a 
particular focal area (24%) and mutual trust across RPP members (20%) were also important for 
maintaining the work of the partnership. 
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Figure 18. Most important conditions for maintaining an RPP, from Phase II survey. Respondents selected 
up to three conditions from a list of 10 options (n = 306 selections). 


Implications 

RPP participants identified mutual organizational interest and trust among partners as key 
ingredients for both launching and maintaining partnerships. Data-sharing agreements, 
individual expertise, and support from organizational leaders were important during the launch 
of a partnership, while regular meetings supported the ongoing work. 


IES may want to consider some of these factors as part of the RFA process, though some are 
easier to discern in an application than others. Providing evidence of mutual organizational 
interest could be documented in a proposal, for instance, as could the planned frequency of 
meetings. IES might request evidence in proposals that the problems identified are important to 
key stakeholders on all sides, and that these stakeholders would be eager to help solve them. Such 
evidence could come from a range of sources, like letters of support that testify to involvement in 
shaping the proposal, reports from focus groups, or even local needs assessments. 
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Promising Strategies for Addressing Key Challenges 


In Phase I, we surfaced central challenges faced by RPPs. In Phase II interviews, we asked 
participants to name strategies they had found effective for overcoming these difficulties. 


Key Findings: 


e Phase I participants described challenges related to three areas: (1) turnover of positions 
for those involved in the partnership, as well as leadership turnover within educational 
organizations more generally; (2) differences in researchers’ and practitioners’ typical 
timelines and/or pace of work; and (3) having active members in the partnership with 
authority to act on the RPP’s findings. 


e To overcome these challenges, participants in Phase II recommended investing in strong, 
trusting relationships, communicating regularly, and adjusting course based on changing 
circumstances. 


In Phase I of the study, participants identified key challenges that they faced related to (1) 
turnover of positions for those involved in the partnership or in the organizations more generally; 
(2) differences in researchers’ and practitioners’ typical timelines and/or pace of work; and (3) 
having the “right people at the table” in terms of active members in the partnership with decision- 
making authority to act on the RPP’s findings. 


In Phase II of the study, we asked participants if they had experienced each of these challenges in 
their partnerships. RPP members then shared strategies for addressing them. Here, we present the 
most prominent strategies identified from 95 interviews with RPP researchers and practitioners. 
It is important to keep in mind that members’ descriptions of the strategies that helped them 
address challenges are retrospective accounts and not contemporaneous observations, which 
might be more useful for understanding how RPPs navigate challenges successfully. 


Turnover in Partnerships and Organizations 

There were 24 RPPs where at least one respondent reported that turnover was a challenge for 
their partnership. Here, turnover often meant people changing positions or leaving the practice 
organization, but it sometimes applied to turnover within the researchers’ organizations as well. 


In times of turnover, open, regular communication with remaining partners was critical (17 
RPPs). Staying in contact through a turnover can help to ensure everyone is on the same page in 
terms of any shifts in partnership goals, roles, and responsibilities as the team adjusts. When new 
members enter into the work, these open lines of communication are important as well. Pausing 
the work to introduce new members and hear their interest and perspective is helpful. Explained 
one researcher: 


What we have found to be effective is that we meet with those people right away, 
and we share with them the goals of the partnership or the implementation. 
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We then spend time listening to that person to find out how those 
partnership goals fit for them, and how they see themselves fitting in. 


The second most prominent strategy for anticipating and coping with turnover was starting with 
multiple partners at various levels within the partner organizations (18 RPPs). Having a range of 
people distributed across levels or departments can allow the project to move forward when faced 
with turnover and help create some stability for partnership work. With several people well-in- 
formed of the goals of the RPP, turnover will be less disruptive because those people can keep the 
work moving forward when someone leaves. 


Another common strategy that respondents discussed was building strong relationships within 
the organizations in order to maintain stability within the partnership. Recommended one re- 
searcher: 


You'll be surprised who ends up leaving for any number of reasons. 
Build those good relationships with as many people as possible. It feels 
like a shotgun approach, but try and get deep and trusting relationships 
with as many folks as possible because the team is not gonna look the 
same, even after two years. 


The trust that emerged from these strong relationships helped create smoother transitions when 
people left the partnership and new people joined. The informal relationships across organiza- 
tions were also valuable when it became necessary to replace partners who had left. 


Difficulty Synchronizing the Timing and Pace of Work 

A second challenge centered on different timelines and paces of work of researchers and 
practitioners. Often, this issue presented itself when researchers needed more time to run 
analyses and/or write reports, while practitioners would have liked to receive this information 
more quickly. However, this was not the only way that asynchronicity emerged. One practitioner 
noted: 


It’s not always that the academic is the slow and steady, and that 

the practitioners want to run, run, run. It’s also the reverse sometimes, 
when the researchers have their work and need that input, and sometimes 
it’s a bad time for the practitioners to get to weigh in. 


The most frequently named strategy to help with issues related to timing was regular 
communication about timelines and deadlines (17 RPPs). Here, participants recommended that 
researchers and practitioners check in on a regular basis to stay abreast of short- and long-term 
work goals, upcoming deadlines, and any emerging challenges that required adjustments. 


Further, partnerships can plan their work together in advance, with an eye to each other's 
calendars, timelines, and rhythms to their work (13 RPPs). Understanding the pacing of their 
partners’ work can help ensure that partnerships’ findings are available at appropriate decision- 
making points, or that partnership work does not become a distraction when there are other 
pressing issues. 
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Participants recommended planning with timelines in mind, early in the grant writing process; 
revisiting and revising the timeline and goals once funding comes in; and checking in about 
timelines on an ongoing basis once the work together begins. 


A third strategy named by participants was to offer quick feedback reports, turned around 

on a more frequent basis. Such feedback involves shorter pieces of analysis, draft versions of 
reports, or tables/charts, without the accompanying text of a finalized journal article. Noted 
one researcher, “Building in quick turnaround kind of analysis into the work can satisfy ... a 
number of questions that [the practice partners] had.” A related strategy here is to consider a 
multi-phased approach to analysis, where researchers provide descriptive findings on a shorter 
timeline, while setting up longer-term goals for more time-intensive analysis. 


Other strategies for navigating timeline issues include: 


¢ Stay flexible with timelines when possible. Sometimes, issues emerged that were 
outside of anyone’ control, and participants reported that flexibility was key to 
navigating these changes. (12 RPPs) 


e When time crunches happen or there are misalignments in calendars, directly and 
immediately consult key partners for advice and insight on how to proceed. (9 RPPs) 


¢ Funda project manager or coordinator to keep an eye on deadlines and timing 
issues. Since there was often a lot going on within the educational agencies, some 
partnerships recommended funding the project manager on the researcher team. (8 
RPPs) 


e Accommodate the needs of practitioners when possible. Even though “asks” from 
practitioner partners were not always aligned to their calendars, several researchers 
noted the value of saying “yes” when last minute requests come in. (5 RPPs) 


Having the Right People at the Table 

Overall, there was no standardized list of individuals or groups who should be involved in a 
partnership in order to act upon the partnerships’ findings or strategies for involving them in 
the work. Instead, having the “right people at the table” was dependent on the nature of the 
problem that the partnership was tackling and the individuals, departments, organizations, or 
levels of systems necessary to respond to the partnerships’ findings. As such, it was important, 
as one researcher said, to “read the context” of the work together to figure out who needed to be 
involved. 


One of the most popular strategies was to include people who have the decision-making authority 
to act on the findings in partnership work early (16 RPPs). For example, in several of the 
partnerships, the co-PI position was held by a leader in the research, assessment, and evaluation 
department of the educational organization—a role that is not always involved in programmatic 
decisions outside of that department. A researcher in one partnership explained how this 
arrangement had constrained the district’s ability to respond to partnership recommendations: 
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The co-PI of the grant on the district side is the head of research and evaluation. 
They do program evaluation. They do assessments. They collect data. ... He really 
feels that he’s not in a position, and he’s right, to take lessons from our work and 
turn them into practice changes for the district. The people who would be the 
right people to implement any practice changes in the district that result from 
our work have not always regularly attended the meetings. ... They’re not 

named per se as key players in the grant and, in a way, they’re not being held 
accountable for the progress of the project on the district side. The person 

who is being held accountable is feeling like he’s not able to implement. 


To avoid a similar situation, partners recommended identifying and reaching out to other 
departments or organizations that have the decision-making authority to make changes aligned 
with partnership findings. It may be best to do this early on in the project (ie., during grant 
development). A subset of participants recommended considering not only who has the authority 
to make decisions but also who has control over relevant budget, staffing, or resource decisions to 
act upon the decisions; these were not necessarily the same individuals or departments. 


A second strategy was to expand participation and engage a broad range of groups that are 
implicated in the problem of practice (16 RPPs). Building a broader coalition was not only helpful 
during the decision-making process but also during implementation of the decision. Insights 
from across stakeholders helped inform and improve the partnerships’ recommendations. A 
broader group also had the added benefit, as noted earlier, of maintaining partnership continuity, 
even during periods of turnover. Depending on the problem of practice, this broader group might 
involve teachers, parents, school leaders, district leaders, school boards, other higher education 
organizations, or state-level policymakers. To identify relevant stakeholders, one participant 
recommended asking practitioners who are knowledgeable about the educational system and 
problem at hand for help in mapping out different stakeholder groups and identifying specific 
people who could be involved. 


A third strategy was to develop relationships with key individuals in the educational agency 

who are well-connected to other stakeholders in the organization (14 RPPs). Identified by one 
researcher as “boundary spanners,’ these individuals helped support partnerships’ ability to move 
on findings. Participants noted that their introductions to others in the educational agency helped 
spread the work, and their support of the project supported the credibility of the findings, making 
them likely to be acted upon. These individuals provided important insight into the priorities 

of the educational agency, helping to make sure that the project was aligned with the other 

work of the agency. Finally, these boundary spanners assisted in translating findings and offered 
suggestions on how and when to share findings, which may have made it more likely they would 
resonate with key decision makers. 


Finally, there was particular value to involving top leadership from the educational organization, 
if possible (10 RPPs). Beyond the fact that these individuals may have had key decision-making 
authority, their involvement in the partnership supported the likelihood of findings being acted 
upon for other reasons. Involving these top leaders—even peripherally—demonstrated that the 
project was a priority for the organization. Top leaders did not always need to be involved in the 
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day-to-day work of the partnership but played an important role as project champions. Their 
involvement may have increased the likelihood for others in the organization to mobilize behind 
it, 


Other strategies named by RPP members included: 


e Consider the range of ways for participants to be involved and/or the possible roles 
that participants may assume. Not everyone needs to be centrally engaged in every 
stage of the work together. (5 RPPs) 


¢ Be patient and respectful and wait for an opportune time to engage. When practice 
partners were focusing on other, more pressing issues, it was difficult to get the “right” 
decision makers at the table. (3 RPPs) 


e From the beginning of the partnership, hold collaboration and mutual decision 
making as a key value to the partnership. It may be easier to engage relevant decision 
makers if the values of mutualism and joint work are clear from the beginning. (2 
RPPs) 


¢ Design the project to be the work of the educational organization or to align with 
other initiatives, rather than as something “on top of” other initiatives. Explained one 
practitioner, “|The partnership] really focused on what we want to do and how we 
want to solve problems, and [the researchers are] just helping us do it. We're able to 
say this is not just a [university] thing. This is a district thing that we are going to do.” 


(2 RPPs) 


e Embed or co-locate a member of the research team within the practice organization. 
Although this was not possible for all RPPs, it helped connect the work of the 
partnership to the day-to-day work of the practice organization, and assisted 
researchers to understand the decision-making process and to learn about key 
decision makers in the practice organization. (2 RPPs) 


Implications 

Many of the challenges named by the IES RPPs are ones that prior research has identified as 
typical of partnerships.* In other words, these challenges are not specific to RPPs in education, 
and they can occur in even the healthiest of partnerships. Consistent with prior research, 
sometimes the challenges impacted the longer-term viability of the partnerships. When the 
challenges were managed successfully, it was because partners engaged in ongoing work to build 
and maintain strong, trusting relationships, they communicated regularly, and they adjusted 
course to adapt to new circumstances. 


In the grant writing process, partnerships could be encouraged to offer initial ideas of how they 
might navigate common challenges, should they arise. For instance, a partnership could lay out 
a set of regular meetings for checking in and reflecting with partners. This early planning would 
provide a foundation for regular communication between partners and help build infrastructure 
that would deepen relationships, support ongoing engagement in the work, and help partners 
navigate future challenges. 
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Conditions that Support Research Use and Partnership Work in 
Practice Organizations 


The current RFA for the IES grant program proposes that partnerships use the two years of grant 
funding to plan for future work together. In this section, we share information regarding the 
reported plans of the participating RPPs for their future partnership work. 


Key Findings: 
e In terms of organizational culture for research use, the majority of RPP practitioners agreed 


that research was seen as a useful source of information in their organizations, but fewer 
indicated that they were expected to back up claims with research in meetings. 


« When compared to a national sample of district leaders, RPP district leaders on average 
reported a less research-oriented organizational culture than did their peers nationwide. 


e Communication may be a challenge in some practice organizations. Only half of respondents 
reported that there were times and places to make sense of new information from partners, or 
that new knowledge was regularly communicated across departments. 


e A majority of practitioners reported that it was easy to see the connections between their 
organizations’ initiatives and their work with external partners. However, two-thirds of RPP 
practitioners reported that organizational leadership did not coordinate work effectively to 
limit conflicts or reduce overlap between organization initiatives and partnership work. 


e Only about half of practitioners agreed or strongly agreed that their organizations had the staff 
and the time needed to plan and implement partnership activities. 


In the Phase II survey, 51 practitioners representing all 27 participating RPPs responded to 
items related to their organization's culture of research use or the conditions associated with the 
organization's capacity to engage productively with an external partner. 


Before exploring these findings, we must first caution about drawing too strong a conclusion 
about the organizational context of the RPP practice agencies based on this analysis alone. We 
had, on average, two to four practitioners from each RPP complete the Phase II survey. They 
were reporting on complex, multi-layered organizations with many departments and dozens or 
hundreds of people working within. These findings help us think about the issues of research 
use or RPP work in organizational terms, but additional work would be needed to more fully 
understand the organizational context of each agency. 


Organizational Culture of Research Use 

A number of scholars have posited that a “culture of research use” is an important condition for 
research use and evidence-based policymaking at the local level. Such a culture is one in which an 
organization’s members value research as a resource for decision making, select strategies using 
evidence, remain open to change in light of evidence, and enact multiple social supports and 
norms promoting evidence use. Further, stronger organizational cultures of research use have 
been found to be associated with greater reports of research use by district and school leaders.” 
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As Figure 19 shows, the majority of RPP practitioners (n = 51) reported that research was seen as 
a useful source of information in their organization often or all of the time (74%). The majority of 
practitioners also responded that they were encouraged to use research often or all of the time in 
their work (64%). However, far fewer indicated that they often or all of the time conducted evalu- 
ations of their programs (38%) or were expected to back up claims with research (29%). 
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research as part of our ongoing work. q 

We conduct studies on programs we 

select and implement to see how they 25% Ea 
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Figure 19. Practitioners’ reports about their organizational culture for research use, from survey (n = 51 RPP 
practitioners). 


We also asked about the organizational culture of research use in school districts in the NCRPP 
national survey of a random subset of subset of school and district leaders (n = 372). Figure 20 
compares the percentage of national survey participants who responded “often” or “all of the 
time” to the percentage of RPP district leaders who responded the same. Because practitioners in 
RPPs held roles in a variety of educational organizations, the comparison sample here is narrowed 
to only those practitioners who held district leadership roles (n = 31). 
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Figure 20. School and district leaders’ perceptions of their organizational culture for research use, from RPP 
survey and national survey (n = 31 RPP district leaders; n = 372 national survey school and district leaders). 


A majority of both samples responded favorably to research being seen as useful and encouraged 
in their organization. Across all items, however, the RPP district leaders had less positive reports 
than did the national sample subset. For instance, over half of national survey respondents (54%) 
indicated that their districts conducted their own studies of their programs “often” or “all of the 
time,” compared to only one-third (33%) of RPP district leaders. 


Conditions that Support an Organization's Capacity to Engage Productively with a 
Partner 

Some organizations may be better positioned to engage with external partners than other orga- 
nizations. The degree to which an organization can engage productively with an external partner 
is related to the organization's “absorptive capacity.’ Absorptive capacity refers to an organiza- 
tion’s ability to recognize the value of new information, assimilate it, and apply it in novel ways.*! 
Research has identified several organizational conditions that support absorptive capacity: com- 
munication pathways, strategic knowledge leadership, and resources for partnering.” Below we 
describe each condition further and report on results from the Phase II survey of RPP practi- 
tioners (n = 51). 
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Communication pathways in the organization. An organization's capacity to act on the knowledge 
gained by working with an external partner may partially depend on the presence of internal 
communication pathways. These are the formal and informal structures within and between de- 
partments that enable people to share, make meaning of, and use knowledge to problem solve. 


Many large, multi-layered organizations face challenges with supporting knowledge-sharing with- 
in and across departments. We see a similar portrait emerge from the reports of the RPP prac- 
titioners about their organizations. As seen in Figure 21, about two-thirds of RPP practitioners 
(64%) agreed or strongly agreed that their organization emphasized cross-department work to 
solve problems that arose from partnership work. Only about half of the respondents, however, 
reported that there were times and places to make sense of new information from the partners 
(55%), or that new knowledge was regularly communicated across departments (51%). 
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from implementing external partnership 
activities. 


There are times and places for us to make sense 
of new information from our external partners. 
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going on in other departments. 
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Figure 21. Practitioner perceptions of their organizational communication pathways, from survey (n = 51 
RPP practitioners). Asterisk (*) indicates item is negatively worded. 


Strategic knowledge leadership. Leaders can play an important role in the degree to which engage- 
ment with an external partner connects to ongoing initiatives. Specifically, “strategic knowledge 
leadership” refers to the ability to identify and assess current sources of knowledge within the 
department, scan the broader field for available sources of knowledge, and synthesize acquired 
knowledge by linking it with current practices or routines. Applied to school districts, strategic 
knowledge leadership could involve efforts to link knowledge from external partners to existing 
improvement efforts. 
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As seen in Figure 22, a large majority (83%) of practitioners reported that it was easy to see 
connections between their organizations’ initiatives and their work with external partners. But 
reports also indicate that this could be difficult to do. Practitioners were split as to whether their 
organizations had the capacity to coordinate work across multiple partners (49%), and only one- 
third (34%) suggested that leaders in their organizations coordinated partnership work to reduce 
conflicts or overlap. 
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Figure 22. Practitioner perceptions of strategic knowledge leadership efforts to connect partnership work 
to ongoing initiatives, from survey (n = 51 RPP practitioners). 


Resources to support partnership activities. Coordinating with external partners can involve a 
great deal of “invisible work.’ It requires resources to support district leaders’ ability to produc- 
tively engage and benefit from working in partnerships. The work of developing and maintaining 
partnerships requires resources that support the time of key stakeholders dedicated to engaging 
with the partners; staff dedicated to partnership coordination; and the purchase of services and 
materials integral to the work. 


Indeed, having the organizational resources to support partnering was limited in the practice 
organizations, based on survey results. Only about half of practitioners (47% to 53%) agreed or 
strongly agreed that their organization had the staff and time needed to plan for and implement 
partnership activities. Further, only one-third (34%) agreed that they had the funding needed to 
plan and implement partnership activities (see Figure 23). 
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Figure 23. Practitioner perceptions of organizational resources to support partnership activities, from 
survey (n = 51 RPP practitioners). 


Implications 

Most RPP practitioners agreed that research was seen as a useful source of information in their 
organizations. Fewer indicated that they were encouraged to conduct evaluations of their pro- 
grams or were expected to back up claims with research in meetings. Somewhat surprisingly, 
across all items, RPP district leaders had less positive reports for organizational culture of re- 
search use than did the national sample subset. It may be that—as co-leaders in RPPs—these 
respondents were strongly oriented to research and perceived their colleagues to be less so. They 
may have been more critical of an organizational culture that supported research use than their 
peers from a more representative sample of leaders. Or, it may be that organizational leaders who 
perceived a relatively weaker culture of research use found it valuable to partner with external 
researchers. It would be worthwhile for future studies to explore how interest or participation in 
RPPs relates to a culture of research use. 


There were also mixed reports about the conditions that support practice organizations’ capaci- 
ties to engage productively with external partners. While most RPP practitioners felt their orga- 
nizations work with external partners supported their goals, they were split as to whether their 
organizations could coordinate the work of various partners or whether they had the resources 
and staffing needed to plan and implement partnership work. In the future, these organizational 
conditions in educational agencies might serve as key points of leverage for practitioners and pol- 
icymakers seeking to increase organizations’ capacities to benefit from research or ideas gained 
through a partnership. 
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RPPs' Plans for Future Work Together 


In our Phase I report, we noted that sustainability for the partnerships past the two-year time 
frame was a concern for many participants. In Phase II, we obtained additional information about 
RPPs’ plans to continue working together and to secure additional funding. 


Key Findings: 
e In Phase II, 16 of the partnerships were still actively working under their IES RPP grants, 
while 11 projects had recently completed their IES RPP grants. 


e The majority of partnerships had continued working together past the end of their IES 
RPP grants or planned to continue to do so. 


e Based on survey reports of PIs and co-Pls, six of the 27 partnerships had successfully 
applied for and received additional funding; another five had applied for additional 
funding but did not receive it. 


e Ten ongoing partnerships had plans to apply for additional funding. The remaining six 
partnerships did not have plans to apply for additional funding at the time of the survey. 


e Partnerships pursued funding from IES, the Spencer Foundation, the National Science 
Foundation, the National Institutes of Health, local and national foundations (e.g., Annie 
Casey Foundation; James McDonnell Foundation; William T. Grant Foundation), and 
state agencies. 


Plans for Future Joint Work 

As reported by project PIs (and by co-Pls in the two cases where the PI for the partnership 

did not complete a survey), IES funding for 16 RPPs was still in progress, whereas funding for 

11 RPPs had recently ended. The majority of PIs/co-PIs indicated that their partnerships had 
continued to work together or planned to continue working together beyond the end of the IES 
RPP grant (see Table 8). Only two PIs whose grants were still in progress were uncertain whether 
their partnerships would continue once the grant ended. 


Table 8. RPPs' Plans for Future Joint Work* 


All Grant Ended Grant in Progress 
n 27 11 16 
Yes 25 hl 14 
No 0 0 0 
Undecided 2 - 2 


* Not a survey option for this category of respondents. 
Note. These data were reported by Pls (or co-Pls if PI did not report). 
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Funding to Continue the Partnership 

We asked participants to report whether they had sought additional funding to continue their 
work together beyond the IES RPP grant (Table 9). Those respondents who indicated that their 
partnership had plans to do so or who had already applied for additional funding were asked to 
name the grant-funding institutions to which they had applied or were planning to apply. Across 
all 27 partnerships, six—approximately one-quarter of RPPs—had successfully applied for and 
received additional funding. 


Table 9. Partnerships’ Funding Plans Beyond IES RPP Grant* 


All Grant Ended Grant in Progress 
n 27 11 16 
Planning to apply for funding, 10 ; 10 


but have not yet done so. 


Submitted an application for 
additional funding, but have 2 - 2 
not yet heard back 


We applied for additional 


funding, and we did receive it. 6 3 3 
We applied for additional 
funding, and we did not re- 5 5 - 
ceive it. 
We did not apply for additional 
: 3 3 O 
funding. 
PI or Co-PI did not report 1 0 1 


*Not a survey option for this category of respondents. 
Note. These data were reported by Pls (or co-Pls if PI did not report). 


Among the partnerships whose grants had already ended (n = 11), three RPPs had applied for 
additional funding and received it; the remaining RPPs had unsuccessfully applied or had not 
applied at all. Among the partnerships whose grants were still in progress (n = 16), three had suc- 
cessfully secured additional funding; 10 had plans to apply for funding but had not yet done so, 
and two others had applied but had not yet heard back. One ongoing partnership did not report 
on plans for future funding, as the PI and co-PI indicated it was uncertain whether the partner- 
ship would continue. 


Often, partnerships submitted grant applications to multiple funding sources. The majority of 
partnerships had pursued or were currently pursuing additional funding from IES (27 RPPs) or 
the Spencer Foundation (17 RPPs) in order to continue their partnership work. Six partnerships 
applied for funding from the National Science Foundation, and three partnerships applied to the 
National Institutes of Health. A total of 14 partnerships applied to other funding institutions, 
including local and national foundations (e.g., Annie Casey Foundation; James S. McDonnell 
Foundation; William T. Grant Foundation), as well as government agencies at the local, state, and 
federal level. 
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Implications 

These results suggest that most of the IES-funded partnerships are interested in continuing 
their work together, and that securing funding to continue partnership work is important to 
RPP members. Though it is evident that securing additional funding can be challenging, it 

is remarkable that more than half of the partnerships (6 of 11) that applied for funding were 
successful in their efforts. This suggests that partnership grants may indeed be a good “on ramp” 
for teams to establish a strong foundation for securing funding for more expansive work. 


Participants reported that IES was the most popular potential source for additional funding. It 
might be helpful, then, for IES to provide guidance to RPP participants on how their work fits 
within the IES goal structure as they look towards future funding. 
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Recommendations and Open Questions 


RPP participants were quite positive about the IES Researcher-Practitioner Partnerships in 
Education Research Program and their experiences with their partners. Their feedback indicates 
that this grant program helped many partnerships build on prior relationships to advance 
ongoing joint work. Participants reported making progress across a range of partnership goals, 
from building a strong relationship to improving student outcomes. At the same time, the RPPs 
faced challenges familiar to partnerships in other sectors, including turnover and differing 
timelines of researchers and practitioners. To overcome these challenges and conditions, partners 
said they fostered more lines of communication, developed multiple relationships with partners 
to mitigate the effects of turnover, and carefully calibrated expectations regarding schedules 

and timetables for completing work. Nearly all individuals who responded to the survey hoped 
to continue their partnership work in the future and, at the time of our data collection, six 
partnerships had secured new funding to do so. The evidence presented here suggests that 
participants found the IES RPP program a valuable means to fostering partnership work focused 
on collaborative research, provided that partners stay nimble in responding to challenges. 


As with our Phase I interim report—and in any study that relies heavily on self-reports—there is 
the potential for bias in these positive evaluations. Two sources of potential bias are noteworthy 
because they relate directly to characteristics of the partnerships and the program itself. First, 
participants may have been overwhelmingly positive about their joint work because these were 
relatively established relationships between researchers and practitioners. All of the partnerships 
had at least some history of working together, and a decision to apply to the RPP program in 

the first place likely signaled a joint commitment to working together. Second, many of the 
partnerships were actively looking for additional funding to continue to support their work 
together at the time of data collection. It makes sense that partners would be positive in their 
reports here, not only because IES provided funding but also because IES is a possible funding 
source in the future. Even so, confidentiality protections helped to ensure that participants felt 
they could be candid in their feedback, and—as evidenced by the willingness of some to share 
negative experiences—we believe they were. Likewise, those who shared positive reports gave 
detailed reasons for doing so in their interviews, which further bolsters the credibility of the 
findings. In sum, our overall findings are warranted through both quantitative and qualitative 
data. 


Below, we offer some specific recommendations to the IES RPP program regarding the structure 
of the grant program, the support offered by IES during the application process, and the RFA and 
resulting proposals. 


Consider a different approach to funding RPPs 

Given the more detailed portrait of prior relationships reported here, we suspect that established 
RPPs were better positioned than new partnerships to secure funding. Reviewing the applications 
of groups that were not funded would help further investigate this hypothesis. If true, the RPP 
program might offer guidance to new applicants to recommend they develop a good working 
relationship via a small project before applying. 
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Similarly, the program could consider a differentiated approach to RPP funding, with different 
goals, timelines, and funding amounts for newer partnerships compared to well-established 
ones. Other grant programs offer differentiated funding structures that may be useful to consider 
for the IES RPP program. For example, IES might consider a structure that supports small, 
medium, and large grants, similar to the NSF’s competition for the Computer Science for All: 
Research+ Practice Partnership program (CS4AlI) held in 2018:* 


e Small proposals (e.g., maximum of $250,000 for up to two years) are designed to 
support the initial steps in establishing a strong and well-integrated RPP team that 
could successfully compete for a medium or large grant in the near future; 


e Medium proposals (e.g., maximum of $600,000 for up to three years) are designed to 
support the modest scaling of a promising approach by a well-defined RPP team; and 


e Large proposals (e.g., maximum of $1,000,000 for up to four years) are designed to 
support the widespread scaling of an evidence-based approach by an RPP team that 
builds on prior collaboration. 


Support-capacity building workshops for prospective grantees 

As with the IES RPP program, the NSF CS4All program aims to support relatively new as well 
as established partnerships. In 2017, it held and funded attendance at two-day workshops for 
prospective teams to help them develop key skills related to RPPs. Of the 21 that were initially 
funded, 19 attended one of the workshops, and attendees reported these to be helpful in their 
preparation for partnership work. IES might consider this approach to help broaden the base of 
applicants to include more successful new partnerships. 


Name the range of goals set by RPPs that have been funded 

We found that participants made progress on a range of goals that extended beyond those clearly 
articulated in the current RFA. The partnerships had ambitious goals to impact local policies 

and practices as well as student outcomes, and they sought to inform the work of others beyond 
their partnerships, including contributing to the knowledge base in education. Further, their 
reported progress provides additional evidence that there are likely “leading” and “lagging” 
indicators for RPPs’ progress on their goals, and it is possible that progress on some goals may be 
interdependent with progress on others. As such, the RPP program may want to consider naming 
in the RFA the range of short- and long-term goals that partnerships have specified in the past, 
without limiting the possible goals that might be pursued by partnerships. 


Broaden the aim of "building capacity to use research" 

Recognizing the multiple ways in which research informs the work of education leaders, the RPP 
program could support evidence use across leaders’ various activities, not just those related to 
adoption decisions but also those that are more common, such as the design of PD. Ultimately, 
RPPs aim to impact student outcomes, but the most direct routes to doing so often involve design 
and testing of innovations. Many RPPs do integrate design into their plan of activities, but this 
could be encouraged explicitly in the RFA in order to impact students more quickly. 
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Ask proposers to present evidence related to conditions that support partnership work 
The RFA could ask for additional information related to the conditions that tend to support a 
partnership's launch or ongoing work, such as mutual organizational interest, evidence of support 
from organizational leaders, the existence of or plans for data-sharing agreements, and plans for a 
regular meeting schedule. For example, IES could request evidence in proposals that the problems 
identified are ones that key stakeholders on all sides agree are important and would be eager to 
help solve. Such evidence could come from a range of sources such as focus groups, local needs 
assessments, or letters of support that testify to involvement in shaping the proposal. 


Invite proposers to consider how they will address anticipated challenges 

The challenges named by RPPs are fairly common ones. In grant proposals or in questions 
submitted to competitive applicants, partnerships could offer initial ideas of how they might 
navigate common challenges, should they come up in the course of their work together. For 
instance, a team could indicate how it is prepared to handle turnover. Or, a partnership could 
share a plan for a set of regular meetings for checking in and reflecting with partners. This early 
planning would provide a foundation for regular communication between partners and help 
build the infrastructure necessary for partners to develop relationships and address challenges 
they may face. 


Further, RPP practitioners offered self-reported strategies for overcoming challenges. It is not 
clear yet, however, when and under what conditions these strategies help mitigate challenges. 
IES may want to further investigate which strategies are associated with more productive and 
successful RPPs. 


Encourage applicants to include a range of professional roles among their members 

It is valuable for practitioners in research roles in their organizations to be included in RPPs. 

At the same time, however, it may also be necessary to involve leaders in other roles in order 

to design policies that can address the partnership's focal problem and influence teaching 

and learning outcomes. In particular, it may be beneficial for RPPs to include individuals 

with decision-making authority related to the problem of practice at hand. For instance, ifa 
partnership is working on a problem of practice related to middle school mathematics, the 
partnership may want to involve directors from the mathematics department as well as assistant 
superintendents who oversee middle schools. The RFA might encourage prospective grantees to 
consider this proactively, so that practitioners involved in the RPP represent different areas of 
the organization that not only bring different perspectives and expertise but also are involved in 
implementation. 


Technical Report No. 3 92 
National Center for Research in Policy and Practice 


Appendices 
Appendix A 
Length of time RPP Partners Had Worked Together Before IES Grant 
Researchers Practitioners 
All Pls Others Co-Pls Others 
n 97 22 of 25 19 
49 
1-3 years (51%) 14 WZ 10 8 
More than 3 years 35 7 12 11 5 
Y (36%) 
Less than 1 year 13 1 2 4 6 
Y (13%) 
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Appendix B 
Most Important Supportive Conditions for Launching the RPP* 


Researchers Practitioners 
All Pls Others Co-Pls Others 
n 332 65 111 84 64 


Mutual organizational interest in the 
focus area (e.g., adult education, 29% 29% 31% 26% 31% 


bilingual education) 


Trust among RPP members 15% 20% 14% 17% 14% 
A data-sharing agreement or 11% 8% 12% 12% 11% 
memorandum of understanding 

Individual expertise 11% 11% 7% 17% 14% 
pes organizational 10% 15% 8% 3% 11% 
Holding regular meetings 9% 0% 10% 10% 6% 
Norms for valuing practitioners and 38% 9% 10% 8% 3% 
researchers’ contribution equally 

Institution support 3% 6% 3% 1% 5% 
Physical proximity of partner 3% 2% 50% 1% 2% 


organizations 


Other 1% 0% 1% 0% 3% 


*Respondents selected the top three conditions from a list of 10 options. 
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Appendix C 
Most Important Supportive Conditions for Maintaining the RPP* 


Researchers Practitioners 
All Pls Others Co-Pls Others 
n 306 65 98 77 66 
Holding regular meetings 25% 20% 29% 26% 23% 


Mutual organizational interest in the 
focus area (e.g., adult education, 24% 23% 23% 23% 27% 


bilingual education) 


Trust among RPP members 20% 29% 19% 17% 15% 
aes organizational 10% 14% 11% 6% 8% 
A data-sharing agreement or 9% 8% 6% 10% 12% 
memorandum of understanding 

Individual expertise 7% 5% 5% 10% 6% 
ae of partner 3% 0% 3% 50% 2% 
Institution support 2% 2% 1% 1% 5% 
Norms for valuing practitioners' and 0% 0% 0% 0% 0% 


researchers’ contribution equally 


Other 1% 0% 2% 0% 3% 


*Respondents selected the top three conditions from a list of 10 options. 
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Appendix D 


IES RPP Contributions to Local Policies or Practices 


Researchers Practitioners 
RPP* Pls Others Co-Pls Others 
n 27 25 37 28 24 
A new framework or set of ideas to think 
about/address focal problem(s) a 16 co 15 16 
Design professional development related to 19 12 4 14 13 


focal problem(s) 


Integration of new practices in the 
educational organization that were 15 12 10 10 10 
developed or modeled in the partnership 


Design of a new practice or program to ad- 


dress focal problem(s) 14 13 17 12 9 


New/revised policy in an 


educational organization a2 ta ¢ _ 


Decision to adopt a new program 
or intervention in an educational 8 10 13 5 5 
organization 
*An RPP was included in each count if at least one practitioner from the partnership answered "yes." 
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Appendix E 
Practitioners' Involvement in Educational Decision-Making Activities, by Percentage of Role-specific 
Group* 
su Practitioners in —_ Practitioners not 
All Practitioners Research Roles _ in Research Roles 
i 51 22 29 
Rane ; 20 7 13 
Purchasing intervention or targeted program (39%) (32%) (459%) 
Directing resources to program ae 13 os 
8 pICE (63%) (59%) (66%) 
Scaling up pilot program 31 it nk 
a (61%) (64%) (59%) 
Eliminating program or polic a 10 = 
a cee (47%) (45%) (48%) 
Designing professional development ot 40 oe 
eae? P (61%) (45%) (48%) 
Redesigning program 20 6 a 
pee (39%) (27%) (48%) 


*Percentages reflect the proportion of each sub-group. 
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Appendix F 
Format of Research Named in Open-Ended Survey Item 
RPP Study National Sample 
All District Research All District Research 
Leaders Roles Leaders Roles 

n 26 15 8 377 307 43 
Journal article 46% 53% 63% 14% 14% 19% 
Research/policy report 38% 27% 13% 17% 20% 26% 
Book 12% 13% 13% 58% 54% 42% 
Research-based tool A% 7% 13% 50% 50% 50% 
or program 
Practitioner-oriented 0% 0% 0% A%, 50% 7% 
magazine article 
las (e.g., webinar, blog, 0% 0% 0% 7% 7% 7% 
Dissertation 0% 0% 0% 1% 1% 0% 
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Appendix G 
Topics and Subtopics of Research Named in Open-Ended Item 


RPP Study National Sample 
District Research District Research 
All All 
Leaders Roles Leaders Roles 
n 36 19 12 552 447 61 
Students and Learning 33% 16% 25% 28% 29% 33% 
a readiness & 50% 67% 33% 3% A% 50% 
penelie aCeue men 25% 33% 67% 61% 63% 42% 
Graduation rates 17% 0% 0% 6% 8% 37% 
Social-emotional outcomes 8% 0% 0% 8% 8% 0% 
oo 0% 0% 0% 16% 15% 16% 
Mindset 0% 0% 0% 5% 2% 0% 
schooborganizasionts 31% 58% 25% 21% 21% 30% 


improvement 


School-level policies & 
interventions (e.g. RTI, 64% 64% 100% 47% 48% 72% 


scheduling, course offerings) 


Bilingual education/English . ‘ ‘ é 3 és 
Language Development (ELD) 18% 18% 0% 8% ie 0% 
Parent/community engagement 9% 9% 0% 5% 6% 6% 
Discipline 9% 9% 0% 7% 8% 6% 
ee leadership 0% 0% 0% 25% 24% 11% 
School climate 0% 0% 0% 8% 6% 6% 
oe & 22% 11% 25% 9% 9% 10% 


System-level policies & 


improvement (e.g., financial aid, 50% 0% 67% 43% 40% 67% 
accountability, school choice, 


organizational change) 
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Appendix G 


Topics and Subtopics of Research Named in Open-Ended Item (continued) 


RPP Study National Sample 
All District Research All District Research 
Leaders Roles Leaders Roles 
n 36 19 12 552 447 61 
Early childhood education 38% 50% 0% 9% 11% 0% 
District/organization- 0% 0% 0% A8% 50% 33% 
level leadership practices 
Teachers and instruction 11% 16% 17% 36% 34% 18% 
Pedagogical strategies 75% 67% 50% 76% 76% 58% 
Curriculum 25% 33% 50% 5% 6% 8% 
Teacher learning (e.g. PLCs, 0% 0% 0% 17% 16% 25% 
professional development) 
Teaches prep? , 0% 0% 0% 3% 2% 8% 
effectivness/evaluation 
Assessment 3% 0% 8% 7% 7% 10% 
Placement 100% NA 100% 32% 33% 17% 
Standardized testing 0% NA 0% 14% 15% 17% 
Classroom assessment 0% NA 0% 41% 36% 67% 
Grading 0% NA 0% 14% 15% 0% 
Note. The n may exceed the number of respondents because some pieces of research addressed multiple 


topics. 
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Appendix H 
Student Subgroups Reflected by Research Named in Open-ended Item 


RPP Study National Sample 
All District Research All District Research 

Leaders Roles Leaders Roles 
n 26 15 8 395 325 45 
None/general 42% 47% 63% 69% 66% 71% 
Special education 23% 40% 13% 7% 7% 4% 
SES/poverty 15% 0% 0% 7% 7% 4% 
Race/ethnicity 8% 7% 13% 8% 9% 9% 
Urban education 8% 7% 13% 2% 2% 11% 
Foster care 4% 0% 0% 0% 0% 0% 
Bilingual students 0% 0% 0% 6% 7% 0% 
Gender or sexual diversity 0% 0% 0% 1% 1% 0% 
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Appendix | 


How Research Named was Useful in Open-Ended items 


n 
Designing policies, programs, 
and initiatives 

Supporting leaders’ own 
professional learning 


Providing instructional 
leadership for others 


Selecting programs 


Supporting and monitoring 
implementation 


RPP Study National Sample 
All District Research All District Research 
Leaders Roles Leaders Roles 
21 13 8 286 236 33 
48% 46% 25% 27% 29% 30% 
24% 23% 13% 29% 29% 33% 
10% 15% 13% 25% 24% 21% 
10% 8% 25% 9% 8% 3% 
10% 8% 25% 9% 10% 12% 
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